Nov 03 22:12:13.290625 ip-10-0-133-193 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:12:13.708799 ip-10-0-133-193 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:13.708799 ip-10-0-133-193 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:12:13.708799 ip-10-0-133-193 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:13.708799 ip-10-0-133-193 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:13.711844 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711762 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:13.711844 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711843 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711848 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711851 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711855 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711858 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711861 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711864 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711866 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711869 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711872 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711875 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711877 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711880 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711883 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711887 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711891 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711895 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711897 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711900 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:13.711910 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711902 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711905 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711908 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711910 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711913 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711916 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711922 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711926 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711928 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711931 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711934 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711936 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711939 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711942 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711944 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711947 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711949 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711952 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711955 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711957 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:13.712362 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711960 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711962 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711965 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711967 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711970 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711973 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711975 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711977 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711980 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711982 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711985 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711987 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711990 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711993 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711996 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.711999 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712001 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712004 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712007 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712010 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:13.712817 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712013 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712016 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712018 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712021 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712024 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712026 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712029 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712031 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712036 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712040 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712044 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712047 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712050 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712052 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712055 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712057 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712060 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712062 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712065 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712068 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:13.713327 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712071 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712074 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712078 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712081 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712085 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712089 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712091 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712094 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712097 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712099 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712102 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712105 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712108 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712111 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712114 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712116 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712119 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712121 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712124 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712126 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:13.713825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712129 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712131 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712134 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712136 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712139 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712141 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712969 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712977 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712981 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712983 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712987 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712990 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712992 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712995 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.712998 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713001 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713004 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713007 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713010 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:13.714321 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713012 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713015 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713019 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713021 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713024 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713026 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713029 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713032 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713034 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713037 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713039 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713042 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713045 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713049 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713052 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713056 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713058 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713061 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713063 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:13.714813 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713066 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713068 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713071 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713074 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713076 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713079 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713081 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713084 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713086 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713090 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713093 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713096 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713099 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713101 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713104 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713107 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713110 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713112 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713115 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713118 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:13.715261 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713121 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713123 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713126 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713128 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713132 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713135 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713138 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713141 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713143 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713146 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713148 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713151 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713153 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713156 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713159 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713161 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713164 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713167 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713169 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:13.715775 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713172 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713174 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713177 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713180 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713183 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713186 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713188 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713191 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713193 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713196 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713198 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713203 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713206 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713208 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713211 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713214 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713216 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713219 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713222 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:13.716222 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713225 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713228 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713230 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713233 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713235 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713238 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713240 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713243 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713246 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713248 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713251 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713254 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713256 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713259 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713261 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713264 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713801 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713808 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713811 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713814 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:13.716680 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713817 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713820 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713823 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713825 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713830 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713833 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713836 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713839 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713841 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713844 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713847 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713850 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713853 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713855 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713858 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713860 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713865 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713867 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713870 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:13.717141 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713872 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713877 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713880 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713884 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713887 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713890 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713893 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713896 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713898 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713901 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713904 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713907 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713909 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713912 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713914 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713917 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713919 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713922 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713925 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:13.717618 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713928 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713930 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713933 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713935 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713938 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713940 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713943 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713945 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713948 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713950 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713954 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713957 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713959 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713962 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713964 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713967 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713969 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713972 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713974 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713977 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:13.718053 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713979 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713982 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713984 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713986 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713989 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713992 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713995 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.713997 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714000 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714003 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714008 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714011 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714013 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714016 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714018 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714021 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714023 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714026 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714028 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714031 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:13.718539 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714033 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714036 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714040 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714042 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714045 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714048 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714050 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714053 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714056 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714060 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714063 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714066 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714069 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714072 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714074 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714077 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714079 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714082 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714086 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:13.719004 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714089 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714091 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714094 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714097 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.714099 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714884 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714898 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714906 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714911 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714915 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714918 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714923 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714928 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714931 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714934 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714938 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714942 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714945 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714948 2580 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714951 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714954 2580 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714957 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714960 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714964 2580 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:12:13.719473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714967 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714970 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714973 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714977 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714981 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714984 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714987 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714991 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714994 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.714998 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715001 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715004 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715007 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715011 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715014 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715017 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715020 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715023 2580 flags.go:64] FLAG: --enable-server="true" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715026 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715031 2580 flags.go:64] FLAG: --event-burst="100" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715034 2580 flags.go:64] FLAG: --event-qps="50" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715038 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715041 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715044 2580 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:12:13.720029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715049 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715052 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715055 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715058 2580 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715061 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715064 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715067 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715070 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715073 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715076 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715079 2580 flags.go:64] FLAG: --feature-gates="" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715083 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715086 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715089 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715092 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715095 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715099 2580 flags.go:64] FLAG: --help="false" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715104 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-133-193.ec2.internal" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715107 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715110 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715113 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715116 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715120 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715123 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:12:13.720629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715126 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715129 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715132 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715135 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715138 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715141 2580 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715144 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715147 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715150 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715153 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715156 2580 flags.go:64] FLAG: --lock-file="" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715159 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715162 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715165 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715171 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715173 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715177 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715179 2580 flags.go:64] FLAG: --logging-format="text" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715182 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715186 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715189 2580 flags.go:64] FLAG: --manifest-url="" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715192 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715196 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715199 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715203 2580 flags.go:64] FLAG: --max-pods="110" Nov 03 22:12:13.721257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715209 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715212 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715215 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715218 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715221 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715224 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715227 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715234 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715237 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715240 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715244 2580 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715246 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715249 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715252 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715255 2580 flags.go:64] FLAG: --port="10250" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715258 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715261 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02ae7526c9f2e8caa" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715264 2580 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715267 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715271 2580 flags.go:64] FLAG: --register-node="true" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715274 2580 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715278 2580 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715281 2580 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715284 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715287 2580 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715301 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:12:13.721888 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715304 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715307 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715310 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715313 2580 flags.go:64] FLAG: --runonce="false" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715316 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715320 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715325 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715328 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715331 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715334 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715337 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715340 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715343 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715346 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715349 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715353 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715357 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715360 2580 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715363 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715368 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715371 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715374 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715378 2580 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715381 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715384 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:12:13.722506 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715387 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715390 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715393 2580 flags.go:64] FLAG: --v="2" Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715402 2580 flags.go:64] FLAG: --version="false" Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715406 2580 flags.go:64] FLAG: --vmodule="" Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715410 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715414 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715548 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715552 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715554 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715557 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715560 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715563 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715567 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715570 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715573 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715576 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715578 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715581 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715584 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715586 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:13.723099 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715589 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715592 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715595 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715598 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715600 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715603 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715606 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715609 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715611 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715614 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715616 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715619 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715621 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715624 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715626 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715629 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715632 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715636 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715639 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:13.723685 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715642 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715645 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715648 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715650 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715653 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715656 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715659 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715661 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715664 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715667 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715669 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715672 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715675 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715677 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715681 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715683 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715686 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715689 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715691 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715693 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:13.724142 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715696 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715699 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715701 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715704 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715706 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715709 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715711 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715714 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715717 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715719 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715722 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715724 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715727 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715729 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715732 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715734 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715737 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715741 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715744 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715746 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:13.724664 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715749 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715752 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715755 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715757 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715760 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715762 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715766 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715769 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715772 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715775 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715777 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715780 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715783 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715787 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715791 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715794 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715796 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715799 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715802 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:13.725151 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715804 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715807 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715809 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715812 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715815 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715818 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715820 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715823 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715825 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715827 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715831 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715834 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715837 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715839 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.715865 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715902 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:13.725623 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715906 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715909 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715913 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715916 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715920 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715925 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715928 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715930 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715933 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715935 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715938 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715941 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715944 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715946 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715949 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715952 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715954 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715957 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715959 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715962 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:13.725995 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715964 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715967 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715969 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715972 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715974 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715977 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715981 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715984 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715986 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715989 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715991 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715994 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715997 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.715999 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716002 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716005 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716008 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716011 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716014 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716017 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:13.726526 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716019 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716022 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716025 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716027 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716030 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716032 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716035 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716037 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716040 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716042 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716045 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716047 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716052 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716056 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716059 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716062 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716065 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716067 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716071 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:13.726996 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716073 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716076 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716078 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716081 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716083 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716086 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716089 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716092 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716095 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716099 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716102 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716104 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716107 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716109 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716112 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716114 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716117 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716119 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716122 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716125 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:13.727443 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716128 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716130 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716133 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716135 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716138 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716141 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716143 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716145 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716148 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716151 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716154 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716158 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716160 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716163 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716165 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716168 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716170 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716173 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716175 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:13.727897 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716178 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716181 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716185 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716187 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716190 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716193 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.716195 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.723320 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.723336 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.723347 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.723352 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723418 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723423 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723426 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723429 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723432 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723435 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723437 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723440 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723443 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:13.728377 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723447 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723449 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723452 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723455 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723457 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723460 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723463 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723465 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723468 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723471 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723473 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723476 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723479 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723481 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723484 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723486 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723489 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723492 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723494 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:13.728849 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723497 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723507 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723511 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723514 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723517 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723521 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723524 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723527 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723530 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723533 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723535 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723538 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723540 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723543 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723546 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723548 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723551 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723553 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723556 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:13.729286 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723558 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723561 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723563 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723566 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723568 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723571 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723573 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723575 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723578 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723580 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723583 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723585 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723588 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723590 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723594 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723602 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723605 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723608 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723610 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723613 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:13.729754 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723616 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723618 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723620 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723623 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723625 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723628 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723631 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723633 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723636 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723638 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723641 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723643 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723646 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723648 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723651 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723653 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723656 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723658 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723661 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:13.730256 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723663 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723665 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723668 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723670 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723673 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723675 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723678 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723681 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723684 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723693 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723696 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723698 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723701 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723704 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723706 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723709 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723711 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723715 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723725 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:13.730809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723728 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.723755 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723963 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723968 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723971 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723974 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723977 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723979 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723982 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723985 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723987 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723990 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723992 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723995 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.723997 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724000 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:13.731253 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724002 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724005 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724007 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724010 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724013 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724016 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724018 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724026 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724029 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724032 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724034 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724037 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724039 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724042 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724044 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724047 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724049 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724052 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724054 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724057 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:13.731645 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724060 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724062 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724065 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724067 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724070 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724072 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724074 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724077 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724079 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724082 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724085 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724087 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724089 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724093 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724097 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724100 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724103 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724105 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724108 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:13.732157 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724110 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724120 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724122 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724125 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724127 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724130 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724132 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724135 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724138 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724140 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724143 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724145 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724148 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724151 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724153 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724156 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724159 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724161 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724164 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724166 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:13.732672 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724169 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724172 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724174 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724176 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724179 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724181 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724184 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724186 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724189 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724191 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724194 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724196 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724199 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724201 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724209 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724213 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724217 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724220 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724223 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:13.733257 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724225 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724228 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724230 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724233 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724235 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724238 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724240 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724243 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724245 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724248 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724250 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724253 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724256 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724258 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.724280 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:13.733728 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724333 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724337 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724340 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724343 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724345 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724348 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724350 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724353 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724356 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724358 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724361 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724363 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724366 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724376 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724380 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724382 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724384 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724387 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724389 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724392 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:13.734082 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724394 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724397 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724399 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724402 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724405 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724407 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724410 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724413 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724415 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724418 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724420 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724423 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724426 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724428 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724431 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724433 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724435 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724439 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724442 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:13.734557 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724445 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724447 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724450 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724452 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724455 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724457 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724460 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724468 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724471 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724473 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724476 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724479 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724481 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724484 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724486 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724489 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724492 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724494 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724496 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:13.735129 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724499 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724502 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724505 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724507 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724510 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724512 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724515 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724517 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724519 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724522 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724524 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724527 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724530 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724532 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724535 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724537 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724540 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724542 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724547 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724550 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:13.735809 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724552 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724560 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724563 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724565 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724568 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724570 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724573 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724575 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724578 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724580 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724583 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724586 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724588 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724591 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724593 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724595 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724598 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724600 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724603 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724605 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:13.736365 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724608 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724611 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724613 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724616 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724620 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724624 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724627 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:13.724630 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.724692 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.727768 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.728684 2580 server.go:1014] "Starting client certificate rotation" Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.728785 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.729347 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.730801 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:12:13.736841 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.736251 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:12:13.746824 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.746806 2580 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:12:13.750709 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.750693 2580 log.go:25] "Validated CRI v1 image API" Nov 03 22:12:13.751553 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.751529 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:12:13.759457 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.759436 2580 fs.go:135] Filesystem UUIDs: map[2decf6e9-2b51-441f-9eed-d592e317b8de:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 85c124c2-90a8-432a-bdbd-e08b0c025042:/dev/nvme0n1p4] Nov 03 22:12:13.759519 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.759456 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 22:12:13.764724 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.764615 2580 manager.go:217] Machine: {Timestamp:2026-11-03 22:12:13.763181568 +0000 UTC m=+0.352237249 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099521 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28442d25d02e6aedbb44e5377b2707 SystemUUID:ec28442d-25d0-2e6a-edbb-44e5377b2707 BootID:59ec0a2a-c43f-4ec6-80e4-9275aef873c6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:3f:f8:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:3f:f8:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:ea:d1:3b:1d:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:12:13.764724 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.764723 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:12:13.764838 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.764802 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:12:13.768203 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.768178 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:12:13.768353 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.768205 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:12:13.768394 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.768357 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:12:13.768394 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.768366 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:12:13.768456 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.768403 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:12:13.768456 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.768421 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:12:13.769814 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.769803 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:12:13.772997 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.772985 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:12:13.773115 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.773105 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:12:13.773918 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.773908 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:12:13.773976 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.773924 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:12:13.773976 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.773942 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:12:13.773976 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.773956 2580 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:12:13.773976 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.773965 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:12:13.774439 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.774426 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:12:13.776917 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.776889 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:12:13.784720 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.784694 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:12:13.785544 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.785531 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:12:13.785591 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.785562 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:12:13.785720 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.785710 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:12:13.785752 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.785726 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:12:13.785752 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.785733 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:12:13.785752 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.785738 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:12:13.785752 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.785744 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:12:13.785752 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.785749 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:12:13.786449 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.786440 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:12:13.786483 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.786449 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:12:13.786483 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.786458 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:12:13.786483 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.786464 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:12:13.786483 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.786472 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:12:13.786483 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.786482 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:12:13.787155 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.787146 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:12:13.787155 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.787155 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:12:13.791036 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.791024 2580 server.go:1263] "Started kubelet" Nov 03 22:12:13.791202 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.791168 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:12:13.791318 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.791174 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:12:13.791318 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.791242 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:12:13.791915 ip-10-0-133-193 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:12:13.792525 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.792508 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:12:13.793220 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.793208 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:12:13.799245 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.799224 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:12:13.799343 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.799233 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:12:13.799495 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.799477 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:13.800020 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800002 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:12:13.800020 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800021 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:12:13.800159 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800140 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:12:13.800159 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800158 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:12:13.800320 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.800281 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 22:12:13.800546 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800519 2580 factory.go:55] Registering systemd factory Nov 03 22:12:13.800546 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800538 2580 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:12:13.800737 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800721 2580 factory.go:153] Registering CRI-O factory Nov 03 22:12:13.800737 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800737 2580 factory.go:223] Registration of the crio container factory successfully Nov 03 22:12:13.800845 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800770 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:12:13.800845 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800785 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:12:13.800845 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800804 2580 factory.go:103] Registering Raw factory Nov 03 22:12:13.800845 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.800819 2580 manager.go:1199] Started watching for new ooms in manager Nov 03 22:12:13.801832 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.801816 2580 manager.go:319] Starting recovery of all containers Nov 03 22:12:13.809420 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.809376 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:12:13.812475 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.812453 2580 manager.go:324] Recovery completed Nov 03 22:12:13.817370 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.817356 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:13.819047 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.819029 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:13.819114 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.819060 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:13.819114 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.819076 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:13.819736 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.819719 2580 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:12:13.819736 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.819733 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:12:13.819876 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.819723 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:13.819876 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.819749 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:12:13.820519 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.820500 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:13.820580 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.820530 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:13.820580 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.820540 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:13.821137 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.821123 2580 policy_none.go:50] "Start" Nov 03 22:12:13.821137 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.821140 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:12:13.821228 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.821149 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:12:13.822324 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.822308 2580 policy_none.go:44] "Start" Nov 03 22:12:13.829184 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.829158 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 22:12:13.829382 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.829352 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-193.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 22:12:13.830417 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.829169 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-193.ec2.internal.18e4a89aa79fb8d0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-193.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-193.ec2.internal,},FirstTimestamp:2026-11-03 22:12:13.790992592 +0000 UTC m=+0.380048273,LastTimestamp:2026-11-03 22:12:13.790992592 +0000 UTC m=+0.380048273,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-193.ec2.internal,}" Nov 03 22:12:13.835599 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.835579 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:12:13.843422 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.843397 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 22:12:13.843422 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.843399 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 22:12:13.843550 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.843398 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 22:12:13.859957 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.859920 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:12:13.862508 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.862495 2580 manager.go:341] "Starting Device Plugin manager" Nov 03 22:12:13.862570 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.862523 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:12:13.862570 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.862522 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6q5s" Nov 03 22:12:13.862570 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.862532 2580 server.go:87] "Starting device plugin registration server" Nov 03 22:12:13.862803 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.862791 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:12:13.862847 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.862805 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:12:13.862908 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.862894 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:12:13.862981 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.862967 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:12:13.862981 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.862979 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:12:13.863559 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.863541 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:12:13.863614 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.863585 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:13.872343 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.872329 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6q5s" Nov 03 22:12:13.888726 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.888634 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-193.ec2.internal.18e4a89aa94bcc09 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-193.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-193.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-193.ec2.internal,},FirstTimestamp:2026-11-03 22:12:13.819046921 +0000 UTC m=+0.408102603,LastTimestamp:2026-11-03 22:12:13.819046921 +0000 UTC m=+0.408102603,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-193.ec2.internal,}" Nov 03 22:12:13.943533 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.943511 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:12:13.943533 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.943537 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:12:13.944668 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.943555 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:12:13.944668 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.943588 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 22:12:13.958128 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.958105 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:13.963903 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.963864 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:13.964718 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.964702 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:13.964792 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.964734 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:13.964792 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.964746 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:13.964792 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.964771 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-193.ec2.internal" Nov 03 22:12:13.985274 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:13.985258 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-193.ec2.internal" Nov 03 22:12:13.985348 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:13.985278 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-193.ec2.internal\": node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:14.041829 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.041664 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:14.043839 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.043823 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal"] Nov 03 22:12:14.043898 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.043873 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:14.044780 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.044765 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:14.044845 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.044794 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:14.044845 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.044806 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:14.046306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.046283 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:14.046437 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.046423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.046473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.046457 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:14.046954 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.046928 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:14.047037 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.046958 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:14.047037 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.046929 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:14.047037 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.046971 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:14.047037 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.046998 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:14.047037 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.047010 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:14.048388 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.048372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.048447 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.048398 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:14.049139 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.049123 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:14.049215 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.049151 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:14.049215 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.049167 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:14.076025 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.076002 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-193.ec2.internal\" not found" node="ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.081759 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.081744 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-193.ec2.internal\" not found" node="ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.142689 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.142665 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:14.201770 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.201741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebb54ef3d211c3a17d214f74e032fea7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.ec2.internal\" (UID: \"ebb54ef3d211c3a17d214f74e032fea7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.201770 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.201769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.201933 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.201788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.243825 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.243773 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:14.302830 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.302803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebb54ef3d211c3a17d214f74e032fea7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.ec2.internal\" (UID: \"ebb54ef3d211c3a17d214f74e032fea7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.302830 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.302821 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebb54ef3d211c3a17d214f74e032fea7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.ec2.internal\" (UID: \"ebb54ef3d211c3a17d214f74e032fea7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.302931 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.302848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.302931 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.302870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.302931 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.302890 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.302931 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.302901 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.344881 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.344861 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:14.380088 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.380066 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.384993 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.384974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.445644 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.445613 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:14.546034 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.545969 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:14.646326 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.646280 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:14.695350 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.695325 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:14.699977 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.699959 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.720835 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.720809 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:14.722696 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.722676 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.728932 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.728916 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 22:12:14.729070 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.729046 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:12:14.729124 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.729046 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 22:12:14.729124 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.729050 2580 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://a267d71b4e6414faf84e06b2ef860b65-0bb1677787aa549a.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.133.193:35804->16.59.14.202:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:14.775000 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.774970 2580 apiserver.go:52] "Watching apiserver" Nov 03 22:12:14.799802 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.799743 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 22:12:14.857653 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.857626 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:12:14.857902 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.857887 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-djcf8","kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal","openshift-cluster-node-tuning-operator/tuned-lnbkf","openshift-image-registry/node-ca-ctz65","openshift-network-diagnostics/network-check-target-g7xxx","kube-system/konnectivity-agent-d4tnh","openshift-dns/node-resolver-7b4rf","openshift-multus/multus-pjwnk","kube-system/global-pull-secret-syncer-vk57x","openshift-multus/network-metrics-daemon-b27xr","openshift-network-operator/iptables-alerter-hmmwm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5","openshift-multus/multus-additional-cni-plugins-nrm7n"] Nov 03 22:12:14.859667 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.859644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:14.860846 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.860825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:14.862144 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.862130 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.863504 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.863486 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:14.864676 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.864663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:14.864747 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.864722 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:14.865830 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.865813 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:14.867274 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.867258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.868564 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.868549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.869860 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.869842 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:14.869938 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.869895 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:14.871645 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.871622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:14.871728 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:14.871685 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:14.873042 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.873020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:14.874254 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.874233 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:14.874566 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.874549 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:14.874657 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.874644 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4tpvw\"" Nov 03 22:12:14.876991 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.876964 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:07:13 +0000 UTC" deadline="2028-08-11 16:47:42.194735418 +0000 UTC" Nov 03 22:12:14.876991 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.876990 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15522h35m27.317748537s" Nov 03 22:12:14.879559 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879533 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wvrhf\"" Nov 03 22:12:14.879658 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879576 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:14.879658 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879612 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mdchn\"" Nov 03 22:12:14.879658 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879632 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:12:14.879658 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879641 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:12:14.879658 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879643 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:14.879897 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879715 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:12:14.879897 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879727 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:14.879897 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879765 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:12:14.879897 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879795 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jwdqh\"" Nov 03 22:12:14.879897 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.879808 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:12:14.880699 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.880355 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:12:14.880699 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.880416 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:12:14.880699 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.880456 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h4jdd\"" Nov 03 22:12:14.880699 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.880489 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:12:14.886323 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.886303 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 22:12:14.892380 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:14.892335 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed587645896ffb2ebb7f46f234764d4.slice/crio-d50feafba7e23b2aa8fb541005ebc7acaa6448643b3bf15b9843323258174be9 WatchSource:0}: Error finding container d50feafba7e23b2aa8fb541005ebc7acaa6448643b3bf15b9843323258174be9: Status 404 returned error can't find the container with id d50feafba7e23b2aa8fb541005ebc7acaa6448643b3bf15b9843323258174be9 Nov 03 22:12:14.892632 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:14.892614 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb54ef3d211c3a17d214f74e032fea7.slice/crio-922f69908bf42654c7cff9ac12ab13fc008bd97a8480ad6253c5e970184b5263 WatchSource:0}: Error finding container 922f69908bf42654c7cff9ac12ab13fc008bd97a8480ad6253c5e970184b5263: Status 404 returned error can't find the container with id 922f69908bf42654c7cff9ac12ab13fc008bd97a8480ad6253c5e970184b5263 Nov 03 22:12:14.895610 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.895592 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9666k" Nov 03 22:12:14.901624 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.899116 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:12:14.902642 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.902625 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:12:14.906160 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906142 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9666k" Nov 03 22:12:14.906246 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-var-lib-kubelet\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.906246 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-dbus\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:14.906387 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-cnibin\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:14.906387 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-slash\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906387 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-var-lib-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906387 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hmn\" (UniqueName: \"kubernetes.io/projected/2ac77e3c-256e-4a95-8106-de9bda8864c5-kube-api-access-j8hmn\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:14.906387 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-systemd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906572 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-ovn\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906572 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-bin\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906572 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1b7053e-001b-4d96-9bf3-1a768e364346-host\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:14.906572 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906492 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-bin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.906572 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-netns\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906572 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906783 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-cnibin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.906783 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906616 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:14.906783 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-log-socket\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906783 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906783 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-k8s-cni-cncf-io\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.906783 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-etc-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.906783 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-kubernetes\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.907066 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-cni-binary-copy\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.907066 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-netns\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.907066 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:14.907066 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/805b7a3b-03f1-4799-a283-235314973feb-host-slash\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:14.907066 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e881588e-ae98-4f04-947d-1a765b8a4e5e-agent-certs\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:14.907066 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.906994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-node-log\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.907066 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907036 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:14.907269 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907090 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrwn\" (UniqueName: \"kubernetes.io/projected/677c391f-bc16-46c8-9439-02a1f3bf934d-kube-api-access-dmrwn\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:14.907269 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-conf\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.907269 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-sys\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.907269 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-host\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.907269 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-hostroot\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.907269 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e881588e-ae98-4f04-947d-1a765b8a4e5e-konnectivity-ca\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:14.907513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:14.907513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-device-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:14.907513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907358 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-modprobe-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.907513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-systemd\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.907513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-kubelet\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.907513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-multus-certs\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.907672 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:14.907672 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907539 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-config\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.907672 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677c391f-bc16-46c8-9439-02a1f3bf934d-tmp-dir\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:14.907672 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:14.907672 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.907672 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-kubelet-config\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:14.907672 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-systemd-units\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.907853 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907709 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.907853 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-netd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.907853 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/677c391f-bc16-46c8-9439-02a1f3bf934d-hosts-file\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:14.907853 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-daemon-config\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.907853 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:14.907853 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/805b7a3b-03f1-4799-a283-235314973feb-iptables-alerter-script\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:14.907853 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c574\" (UniqueName: \"kubernetes.io/projected/805b7a3b-03f1-4799-a283-235314973feb-kube-api-access-2c574\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:14.908039 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9lch\" (UniqueName: \"kubernetes.io/projected/8fc6de3b-56b8-4344-b7ab-3ad21f645803-kube-api-access-v9lch\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.908039 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9hd\" (UniqueName: \"kubernetes.io/projected/c1b7053e-001b-4d96-9bf3-1a768e364346-kube-api-access-9t9hd\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:14.908039 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-kubelet\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.908039 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.907965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovn-node-metrics-cert\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.908039 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987t6\" (UniqueName: \"kubernetes.io/projected/44c8b7d7-71ad-47ee-9aad-be7f366974f5-kube-api-access-987t6\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.908039 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-socket-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:14.908249 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-sys-fs\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:14.908249 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-socket-dir-parent\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.908249 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908164 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:14.908249 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-script-lib\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.908249 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-registration-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:14.908487 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysconfig\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.908487 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-system-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.908487 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908362 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-multus\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.908487 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-conf-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.908487 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759px\" (UniqueName: \"kubernetes.io/projected/a2dea624-bc31-43e7-8691-00c686c3eacc-kube-api-access-759px\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.908487 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1b7053e-001b-4d96-9bf3-1a768e364346-serviceca\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:14.908487 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-env-overrides\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:14.908487 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rhtc\" (UniqueName: \"kubernetes.io/projected/bf8fa970-77b7-4f41-9a2d-a586343c51da-kube-api-access-7rhtc\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:14.908851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-tuned\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.908851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908575 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-system-cni-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:14.908851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.908851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-run\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.908851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-lib-modules\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.908851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:14.908851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-tmp\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:14.908851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-etc-kubernetes\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:14.908851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.908742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ftn6\" (UniqueName: \"kubernetes.io/projected/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-kube-api-access-5ftn6\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:14.947207 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.947143 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" event={"ID":"fed587645896ffb2ebb7f46f234764d4","Type":"ContainerStarted","Data":"d50feafba7e23b2aa8fb541005ebc7acaa6448643b3bf15b9843323258174be9"} Nov 03 22:12:14.948135 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.948118 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" event={"ID":"ebb54ef3d211c3a17d214f74e032fea7","Type":"ContainerStarted","Data":"922f69908bf42654c7cff9ac12ab13fc008bd97a8480ad6253c5e970184b5263"} Nov 03 22:12:14.971723 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.971697 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:12:14.971841 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.971789 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:12:14.971896 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.971880 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:12:14.971950 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.971914 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:12:14.972029 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972016 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wf9zk\"" Nov 03 22:12:14.972128 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972114 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:12:14.972128 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972124 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:12:14.972226 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972147 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:12:14.972226 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972166 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:12:14.972226 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972167 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:12:14.972226 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972149 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:12:14.972226 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972174 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:12:14.972226 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972207 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:12:14.972473 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972365 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nt8s4\"" Nov 03 22:12:14.972523 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972489 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:12:14.972646 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972632 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:12:14.972682 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972637 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:12:14.972722 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972234 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7shc8\"" Nov 03 22:12:14.972813 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:14.972801 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz46p\"" Nov 03 22:12:15.009168 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1b7053e-001b-4d96-9bf3-1a768e364346-serviceca\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:15.009168 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-env-overrides\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.009356 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhtc\" (UniqueName: \"kubernetes.io/projected/bf8fa970-77b7-4f41-9a2d-a586343c51da-kube-api-access-7rhtc\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.009356 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-tuned\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009419 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-system-cni-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.009452 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009494 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-run\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009494 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-lib-modules\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009576 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.009576 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009512 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-system-cni-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.009576 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-tmp\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009576 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-etc-kubernetes\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.009576 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftn6\" (UniqueName: \"kubernetes.io/projected/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-kube-api-access-5ftn6\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-var-lib-kubelet\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-dbus\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009601 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-run\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009626 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009530 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009651 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-etc-kubernetes\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009665 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-cnibin\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-cnibin\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009690 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-lib-modules\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-slash\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-var-lib-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009729 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-var-lib-kubelet\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hmn\" (UniqueName: \"kubernetes.io/projected/2ac77e3c-256e-4a95-8106-de9bda8864c5-kube-api-access-j8hmn\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-systemd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009777 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-slash\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.009775 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-ovn\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-bin\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1b7053e-001b-4d96-9bf3-1a768e364346-host\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009830 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-dbus\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009846 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-systemd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009862 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1b7053e-001b-4d96-9bf3-1a768e364346-host\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009862 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-var-lib-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009871 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-ovn\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009888 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-bin\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-bin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-netns\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-cnibin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-bin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.009988 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-netns\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-log-socket\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010005 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.010466 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010016 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-cnibin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010056 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010102 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-log-socket\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-k8s-cni-cncf-io\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-etc-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.010183 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010216 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-etc-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010183 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-k8s-cni-cncf-io\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010236 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-kubernetes\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.010247 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret podName:05ab17d1-b5dc-4cb6-bf8e-eb650c724aef nodeName:}" failed. No retries permitted until 2026-11-03 22:12:15.510225525 +0000 UTC m=+2.099281208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret") pod "global-pull-secret-syncer-vk57x" (UID: "05ab17d1-b5dc-4cb6-bf8e-eb650c724aef") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-kubernetes\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-cni-binary-copy\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-netns\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010411 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-env-overrides\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010418 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1b7053e-001b-4d96-9bf3-1a768e364346-serviceca\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:15.011306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/805b7a3b-03f1-4799-a283-235314973feb-host-slash\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e881588e-ae98-4f04-947d-1a765b8a4e5e-agent-certs\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010501 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-node-log\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010506 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-netns\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrwn\" (UniqueName: \"kubernetes.io/projected/677c391f-bc16-46c8-9439-02a1f3bf934d-kube-api-access-dmrwn\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-conf\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-sys\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-host\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-hostroot\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e881588e-ae98-4f04-947d-1a765b8a4e5e-konnectivity-ca\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-device-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-modprobe-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010715 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-systemd\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-kubelet\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-multus-certs\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.012002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-config\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677c391f-bc16-46c8-9439-02a1f3bf934d-tmp-dir\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010886 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-hostroot\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-kubelet-config\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-systemd-units\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010940 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-kubelet-config\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010940 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-cni-binary-copy\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.010994 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011001 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-multus-certs\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011054 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011095 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-kubelet\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011136 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-device-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011149 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-modprobe-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.012762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011203 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-systemd\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011281 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011346 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-sys\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011348 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677c391f-bc16-46c8-9439-02a1f3bf934d-tmp-dir\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011455 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/805b7a3b-03f1-4799-a283-235314973feb-host-slash\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011465 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-conf\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011491 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011531 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e881588e-ae98-4f04-947d-1a765b8a4e5e-konnectivity-ca\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011548 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-host\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-netd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011592 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-systemd-units\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/677c391f-bc16-46c8-9439-02a1f3bf934d-hosts-file\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011638 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-node-log\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011685 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/677c391f-bc16-46c8-9439-02a1f3bf934d-hosts-file\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011697 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-netd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011703 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-daemon-config\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011745 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.013631 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.011879 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.012268 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:15.512249844 +0000 UTC m=+2.101305521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012173 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-config\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.011741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/805b7a3b-03f1-4799-a283-235314973feb-iptables-alerter-script\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c574\" (UniqueName: \"kubernetes.io/projected/805b7a3b-03f1-4799-a283-235314973feb-kube-api-access-2c574\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9lch\" (UniqueName: \"kubernetes.io/projected/8fc6de3b-56b8-4344-b7ab-3ad21f645803-kube-api-access-v9lch\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9hd\" (UniqueName: \"kubernetes.io/projected/c1b7053e-001b-4d96-9bf3-1a768e364346-kube-api-access-9t9hd\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-kubelet\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovn-node-metrics-cert\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987t6\" (UniqueName: \"kubernetes.io/projected/44c8b7d7-71ad-47ee-9aad-be7f366974f5-kube-api-access-987t6\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012585 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-socket-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-sys-fs\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-socket-dir-parent\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012695 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-daemon-config\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.012774 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-kubelet\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.014502 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.013082 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-socket-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.013159 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-sys-fs\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.013581 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-tuned\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.013611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.013658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-script-lib\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.013685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-registration-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.013938 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e881588e-ae98-4f04-947d-1a765b8a4e5e-agent-certs\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.013660 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-socket-dir-parent\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.014048 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-registration-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.013716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysconfig\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.014154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-system-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.014183 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-multus\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.014207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-conf-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.014235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759px\" (UniqueName: \"kubernetes.io/projected/a2dea624-bc31-43e7-8691-00c686c3eacc-kube-api-access-759px\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.014561 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysconfig\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.014621 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-system-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.014670 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-multus\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.015216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.014716 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-conf-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.016034 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.015167 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-script-lib\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.016034 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.015202 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.016563 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.016510 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovn-node-metrics-cert\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.016917 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.016892 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/805b7a3b-03f1-4799-a283-235314973feb-iptables-alerter-script\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:15.017002 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.016982 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-tmp\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.031778 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.031755 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhtc\" (UniqueName: \"kubernetes.io/projected/bf8fa970-77b7-4f41-9a2d-a586343c51da-kube-api-access-7rhtc\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.034435 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.034417 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hmn\" (UniqueName: \"kubernetes.io/projected/2ac77e3c-256e-4a95-8106-de9bda8864c5-kube-api-access-j8hmn\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.037403 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.037378 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftn6\" (UniqueName: \"kubernetes.io/projected/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-kube-api-access-5ftn6\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:15.037496 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.037383 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:15.037496 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.037427 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:15.037496 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.037439 2580 projected.go:196] Error preparing data for projected volume kube-api-access-6sp7f for pod openshift-network-diagnostics/network-check-target-g7xxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:15.037496 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.037490 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f podName:019bf19b-ca0e-4a02-af50-d6450151540e nodeName:}" failed. No retries permitted until 2026-11-03 22:12:15.537475134 +0000 UTC m=+2.126530823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6sp7f" (UniqueName: "kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f") pod "network-check-target-g7xxx" (UID: "019bf19b-ca0e-4a02-af50-d6450151540e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:15.038759 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.038734 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c574\" (UniqueName: \"kubernetes.io/projected/805b7a3b-03f1-4799-a283-235314973feb-kube-api-access-2c574\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:15.039420 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.039394 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9hd\" (UniqueName: \"kubernetes.io/projected/c1b7053e-001b-4d96-9bf3-1a768e364346-kube-api-access-9t9hd\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:15.039697 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.039678 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-759px\" (UniqueName: \"kubernetes.io/projected/a2dea624-bc31-43e7-8691-00c686c3eacc-kube-api-access-759px\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.039817 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.039800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrwn\" (UniqueName: \"kubernetes.io/projected/677c391f-bc16-46c8-9439-02a1f3bf934d-kube-api-access-dmrwn\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:15.040009 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.039992 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-987t6\" (UniqueName: \"kubernetes.io/projected/44c8b7d7-71ad-47ee-9aad-be7f366974f5-kube-api-access-987t6\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.040101 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.040088 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9lch\" (UniqueName: \"kubernetes.io/projected/8fc6de3b-56b8-4344-b7ab-3ad21f645803-kube-api-access-v9lch\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.083964 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.083899 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:15.202820 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.202786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:15.207884 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.207868 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:15.211007 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:15.210985 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805b7a3b_03f1_4799_a283_235314973feb.slice/crio-5f2f55b8f9921029a65e5aa9814552ecefb8a64bf4564f6ce296a77d21718518 WatchSource:0}: Error finding container 5f2f55b8f9921029a65e5aa9814552ecefb8a64bf4564f6ce296a77d21718518: Status 404 returned error can't find the container with id 5f2f55b8f9921029a65e5aa9814552ecefb8a64bf4564f6ce296a77d21718518 Nov 03 22:12:15.214308 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.214272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:15.215563 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:15.215544 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677c391f_bc16_46c8_9439_02a1f3bf934d.slice/crio-0df8cc8c1090d72466ae22675707c566e54249f1899243b79363e6eca1a92334 WatchSource:0}: Error finding container 0df8cc8c1090d72466ae22675707c566e54249f1899243b79363e6eca1a92334: Status 404 returned error can't find the container with id 0df8cc8c1090d72466ae22675707c566e54249f1899243b79363e6eca1a92334 Nov 03 22:12:15.218715 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.218700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:15.221840 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:15.221820 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc6de3b_56b8_4344_b7ab_3ad21f645803.slice/crio-3130b416ab859305547500641fed353d5cdea06316def7e58219e01201653fea WatchSource:0}: Error finding container 3130b416ab859305547500641fed353d5cdea06316def7e58219e01201653fea: Status 404 returned error can't find the container with id 3130b416ab859305547500641fed353d5cdea06316def7e58219e01201653fea Nov 03 22:12:15.223857 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.223836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:15.226150 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:15.226133 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b7053e_001b_4d96_9bf3_1a768e364346.slice/crio-b53ac119811039de567e6ea6c10de3dea98d2d45595c38fc0371e4085ca7af42 WatchSource:0}: Error finding container b53ac119811039de567e6ea6c10de3dea98d2d45595c38fc0371e4085ca7af42: Status 404 returned error can't find the container with id b53ac119811039de567e6ea6c10de3dea98d2d45595c38fc0371e4085ca7af42 Nov 03 22:12:15.231042 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:15.231013 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode881588e_ae98_4f04_947d_1a765b8a4e5e.slice/crio-60a0060ae6f7a4d82e977cd8dea8ea73a2c80457a95a24ddc4acf88a682a2172 WatchSource:0}: Error finding container 60a0060ae6f7a4d82e977cd8dea8ea73a2c80457a95a24ddc4acf88a682a2172: Status 404 returned error can't find the container with id 60a0060ae6f7a4d82e977cd8dea8ea73a2c80457a95a24ddc4acf88a682a2172 Nov 03 22:12:15.237534 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.237514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pjwnk" Nov 03 22:12:15.238625 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.238603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:15.243837 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.243793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:15.246899 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:15.246875 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dea624_bc31_43e7_8691_00c686c3eacc.slice/crio-65ebbaf992317bd2929513c86b7b6287bb273fd789194acec14e6381404a127c WatchSource:0}: Error finding container 65ebbaf992317bd2929513c86b7b6287bb273fd789194acec14e6381404a127c: Status 404 returned error can't find the container with id 65ebbaf992317bd2929513c86b7b6287bb273fd789194acec14e6381404a127c Nov 03 22:12:15.251332 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.251154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:15.256825 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:15.253919 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c8b7d7_71ad_47ee_9aad_be7f366974f5.slice/crio-ec3ddd1cfa9e74d5b968c59a676d4beae48b19ece3bb527da0d1d619dbf8d738 WatchSource:0}: Error finding container ec3ddd1cfa9e74d5b968c59a676d4beae48b19ece3bb527da0d1d619dbf8d738: Status 404 returned error can't find the container with id ec3ddd1cfa9e74d5b968c59a676d4beae48b19ece3bb527da0d1d619dbf8d738 Nov 03 22:12:15.257212 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:15.257185 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8fa970_77b7_4f41_9a2d_a586343c51da.slice/crio-25a25c5c4a88c1d74521e00b53636f628730d545ff52cb48efa3c693550472c6 WatchSource:0}: Error finding container 25a25c5c4a88c1d74521e00b53636f628730d545ff52cb48efa3c693550472c6: Status 404 returned error can't find the container with id 25a25c5c4a88c1d74521e00b53636f628730d545ff52cb48efa3c693550472c6 Nov 03 22:12:15.400895 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.400821 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:15.518024 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.517984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:15.518196 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.518059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:15.518196 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.518148 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:15.518308 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.518221 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:16.518208001 +0000 UTC m=+3.107263691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:15.518308 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.518157 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:15.518308 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.518280 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret podName:05ab17d1-b5dc-4cb6-bf8e-eb650c724aef nodeName:}" failed. No retries permitted until 2026-11-03 22:12:16.51826373 +0000 UTC m=+3.107319418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret") pod "global-pull-secret-syncer-vk57x" (UID: "05ab17d1-b5dc-4cb6-bf8e-eb650c724aef") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:15.619776 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.619134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:15.619776 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.619324 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:15.619776 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.619342 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:15.619776 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.619355 2580 projected.go:196] Error preparing data for projected volume kube-api-access-6sp7f for pod openshift-network-diagnostics/network-check-target-g7xxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:15.619776 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:15.619415 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f podName:019bf19b-ca0e-4a02-af50-d6450151540e nodeName:}" failed. No retries permitted until 2026-11-03 22:12:16.619397893 +0000 UTC m=+3.208453562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sp7f" (UniqueName: "kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f") pod "network-check-target-g7xxx" (UID: "019bf19b-ca0e-4a02-af50-d6450151540e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:15.836271 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.836091 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:15.907993 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.907886 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:07:14 +0000 UTC" deadline="2028-08-03 08:59:51.377720262 +0000 UTC" Nov 03 22:12:15.907993 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.907940 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15322h47m35.469784148s" Nov 03 22:12:15.961160 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.961122 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7b4rf" event={"ID":"677c391f-bc16-46c8-9439-02a1f3bf934d","Type":"ContainerStarted","Data":"0df8cc8c1090d72466ae22675707c566e54249f1899243b79363e6eca1a92334"} Nov 03 22:12:15.965461 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.964239 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hmmwm" event={"ID":"805b7a3b-03f1-4799-a283-235314973feb","Type":"ContainerStarted","Data":"5f2f55b8f9921029a65e5aa9814552ecefb8a64bf4564f6ce296a77d21718518"} Nov 03 22:12:15.982831 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.982756 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pjwnk" event={"ID":"a2dea624-bc31-43e7-8691-00c686c3eacc","Type":"ContainerStarted","Data":"65ebbaf992317bd2929513c86b7b6287bb273fd789194acec14e6381404a127c"} Nov 03 22:12:15.995909 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:15.995876 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d4tnh" event={"ID":"e881588e-ae98-4f04-947d-1a765b8a4e5e","Type":"ContainerStarted","Data":"60a0060ae6f7a4d82e977cd8dea8ea73a2c80457a95a24ddc4acf88a682a2172"} Nov 03 22:12:16.013949 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.013873 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctz65" event={"ID":"c1b7053e-001b-4d96-9bf3-1a768e364346","Type":"ContainerStarted","Data":"b53ac119811039de567e6ea6c10de3dea98d2d45595c38fc0371e4085ca7af42"} Nov 03 22:12:16.021968 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.021609 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" event={"ID":"8fc6de3b-56b8-4344-b7ab-3ad21f645803","Type":"ContainerStarted","Data":"3130b416ab859305547500641fed353d5cdea06316def7e58219e01201653fea"} Nov 03 22:12:16.025598 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.025175 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" event={"ID":"2ac77e3c-256e-4a95-8106-de9bda8864c5","Type":"ContainerStarted","Data":"b4aad4e5a91a2b20a8df1381f665061481cbf65bd6a11b7cb909599fed022247"} Nov 03 22:12:16.026936 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.026637 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" event={"ID":"bf8fa970-77b7-4f41-9a2d-a586343c51da","Type":"ContainerStarted","Data":"25a25c5c4a88c1d74521e00b53636f628730d545ff52cb48efa3c693550472c6"} Nov 03 22:12:16.028672 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.028649 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerStarted","Data":"ec3ddd1cfa9e74d5b968c59a676d4beae48b19ece3bb527da0d1d619dbf8d738"} Nov 03 22:12:16.526366 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.526329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:16.526542 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.526401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:16.526619 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.526545 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:16.526619 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.526610 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret podName:05ab17d1-b5dc-4cb6-bf8e-eb650c724aef nodeName:}" failed. No retries permitted until 2026-11-03 22:12:18.526591418 +0000 UTC m=+5.115647109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret") pod "global-pull-secret-syncer-vk57x" (UID: "05ab17d1-b5dc-4cb6-bf8e-eb650c724aef") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:16.527016 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.526997 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:16.527096 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.527058 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:18.527042565 +0000 UTC m=+5.116098233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:16.627633 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.627601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:16.627810 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.627773 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:16.627810 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.627802 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:16.627921 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.627816 2580 projected.go:196] Error preparing data for projected volume kube-api-access-6sp7f for pod openshift-network-diagnostics/network-check-target-g7xxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:16.627921 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.627878 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f podName:019bf19b-ca0e-4a02-af50-d6450151540e nodeName:}" failed. No retries permitted until 2026-11-03 22:12:18.627860746 +0000 UTC m=+5.216916429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sp7f" (UniqueName: "kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f") pod "network-check-target-g7xxx" (UID: "019bf19b-ca0e-4a02-af50-d6450151540e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:16.908194 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.908099 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:07:14 +0000 UTC" deadline="2028-04-20 13:30:50.174221773 +0000 UTC" Nov 03 22:12:16.908194 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.908136 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12807h18m33.266089691s" Nov 03 22:12:16.944219 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.944185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:16.944387 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.944338 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:16.944756 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.944736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:16.944883 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.944842 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:16.944950 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:16.944911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:16.945007 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:16.944992 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:17.906407 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:17.906340 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:18.546420 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:18.543980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:18.546420 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:18.544060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:18.546420 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.544182 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:18.546420 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.544242 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:22.54422502 +0000 UTC m=+9.133280705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:18.546420 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.544468 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:18.546420 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.544510 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret podName:05ab17d1-b5dc-4cb6-bf8e-eb650c724aef nodeName:}" failed. No retries permitted until 2026-11-03 22:12:22.544496761 +0000 UTC m=+9.133552445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret") pod "global-pull-secret-syncer-vk57x" (UID: "05ab17d1-b5dc-4cb6-bf8e-eb650c724aef") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:18.644525 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:18.644479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:18.644655 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.644625 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:18.644655 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.644643 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:18.644758 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.644656 2580 projected.go:196] Error preparing data for projected volume kube-api-access-6sp7f for pod openshift-network-diagnostics/network-check-target-g7xxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:18.644758 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.644708 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f podName:019bf19b-ca0e-4a02-af50-d6450151540e nodeName:}" failed. No retries permitted until 2026-11-03 22:12:22.644690667 +0000 UTC m=+9.233746352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sp7f" (UniqueName: "kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f") pod "network-check-target-g7xxx" (UID: "019bf19b-ca0e-4a02-af50-d6450151540e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:18.947045 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:18.945835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:18.947045 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.945924 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:18.947045 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:18.945979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:18.947045 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.946046 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:18.947045 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:18.946102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:18.947045 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:18.946178 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:19.050314 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:19.050235 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" event={"ID":"2ac77e3c-256e-4a95-8106-de9bda8864c5","Type":"ContainerStarted","Data":"88d936261865f83acfca416fadcd5ba2512fdebde0febf07d11b9d260c75fbc8"} Nov 03 22:12:19.053738 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:19.053702 2580 generic.go:356] "Generic (PLEG): container finished" podID="fed587645896ffb2ebb7f46f234764d4" containerID="8d8a2255e7c00795dbf578ba6ab3056676898a73cd66483b6559dc8d65bd706c" exitCode=0 Nov 03 22:12:19.054087 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:19.053989 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:19.054087 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:19.054012 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" event={"ID":"fed587645896ffb2ebb7f46f234764d4","Type":"ContainerDied","Data":"8d8a2255e7c00795dbf578ba6ab3056676898a73cd66483b6559dc8d65bd706c"} Nov 03 22:12:19.057527 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:19.057499 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" event={"ID":"ebb54ef3d211c3a17d214f74e032fea7","Type":"ContainerStarted","Data":"78966bec0dd507429c9b3356ffce8d331884270058b776ca5d4ed8bfbdaf024b"} Nov 03 22:12:19.080013 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:19.079968 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:19.081598 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:19.081576 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal"] Nov 03 22:12:19.167366 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:19.167251 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" podStartSLOduration=5.1672321 podStartE2EDuration="5.1672321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:19.167033747 +0000 UTC m=+5.756089438" watchObservedRunningTime="2026-11-03 22:12:19.1672321 +0000 UTC m=+5.756287834" Nov 03 22:12:20.087568 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:20.086859 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" event={"ID":"fed587645896ffb2ebb7f46f234764d4","Type":"ContainerStarted","Data":"c3470596bd9ad98450e02417e42c2958225ff8cb0e89650ad5adec23afe65824"} Nov 03 22:12:20.090452 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:20.090416 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d4tnh" event={"ID":"e881588e-ae98-4f04-947d-1a765b8a4e5e","Type":"ContainerStarted","Data":"e00a015c84934d8c7a6cdcdc2f09f28abe992762fb553cf2dfc9aed94cc37f6b"} Nov 03 22:12:20.094183 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:20.094152 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="88d936261865f83acfca416fadcd5ba2512fdebde0febf07d11b9d260c75fbc8" exitCode=0 Nov 03 22:12:20.094886 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:20.094447 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" event={"ID":"2ac77e3c-256e-4a95-8106-de9bda8864c5","Type":"ContainerDied","Data":"88d936261865f83acfca416fadcd5ba2512fdebde0febf07d11b9d260c75fbc8"} Nov 03 22:12:20.128239 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:20.127752 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" podStartSLOduration=1.127737236 podStartE2EDuration="1.127737236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:20.127147628 +0000 UTC m=+6.716203319" watchObservedRunningTime="2026-11-03 22:12:20.127737236 +0000 UTC m=+6.716792926" Nov 03 22:12:20.169763 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:20.169387 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d4tnh" podStartSLOduration=1.872000843 podStartE2EDuration="6.169369193s" totalImagesPullingTime="4.29736835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:20.168384817 +0000 UTC m=+6.757440508" watchObservedRunningTime="2026-11-03 22:12:20.169369193 +0000 UTC m=+6.758424878" Nov 03 22:12:20.944433 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:20.944395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:20.944613 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:20.944544 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:20.944970 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:20.944945 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:20.945141 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:20.945045 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:20.945141 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:20.945100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:20.945250 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:20.945168 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:21.100144 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:21.100108 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctz65" event={"ID":"c1b7053e-001b-4d96-9bf3-1a768e364346","Type":"ContainerStarted","Data":"8516bac52911631db63cfa3cf7ab2a46d31c4c11c4b50a05112e27e895552224"} Nov 03 22:12:22.106032 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.105996 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" event={"ID":"bf8fa970-77b7-4f41-9a2d-a586343c51da","Type":"ContainerStarted","Data":"fda8b3de7e7a8339593fed5b59d50d567ae6a631ce76585a3c684bf6570e73d6"} Nov 03 22:12:22.506004 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.505092 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:22.506004 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.505823 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:22.547133 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.547077 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ctz65" podStartSLOduration=3.354333897 podStartE2EDuration="8.54705959s" totalImagesPullingTime="5.192725693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:21.141203854 +0000 UTC m=+7.730259546" watchObservedRunningTime="2026-11-03 22:12:22.54705959 +0000 UTC m=+9.136115281" Nov 03 22:12:22.581772 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.580490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:22.581772 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.580592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:22.581772 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.580746 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:22.581772 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.580800 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:30.580783536 +0000 UTC m=+17.169839220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:22.581772 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.581153 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:22.581772 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.581197 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret podName:05ab17d1-b5dc-4cb6-bf8e-eb650c724aef nodeName:}" failed. No retries permitted until 2026-11-03 22:12:30.581183789 +0000 UTC m=+17.170239477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret") pod "global-pull-secret-syncer-vk57x" (UID: "05ab17d1-b5dc-4cb6-bf8e-eb650c724aef") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:22.681910 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.681873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:22.682077 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.682045 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:22.682077 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.682067 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:22.682137 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.682079 2580 projected.go:196] Error preparing data for projected volume kube-api-access-6sp7f for pod openshift-network-diagnostics/network-check-target-g7xxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:22.682180 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.682137 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f podName:019bf19b-ca0e-4a02-af50-d6450151540e nodeName:}" failed. No retries permitted until 2026-11-03 22:12:30.682119485 +0000 UTC m=+17.271175156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sp7f" (UniqueName: "kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f") pod "network-check-target-g7xxx" (UID: "019bf19b-ca0e-4a02-af50-d6450151540e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:22.705520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.705479 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:12:22.867738 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.867623 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:12:22.705502614Z","UUID":"6050e4c8-0413-4672-8741-0a65f9b7a768","Handler":null,"Name":"","Endpoint":""} Nov 03 22:12:22.869543 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.869522 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:12:22.869543 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.869549 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:12:22.944439 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.944371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:22.944439 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.944393 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:22.944439 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:22.944371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:22.944682 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.944521 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:22.944682 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.944552 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:22.944682 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:22.944613 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:23.110251 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:23.110213 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pjwnk" event={"ID":"a2dea624-bc31-43e7-8691-00c686c3eacc","Type":"ContainerStarted","Data":"e8c85cbf8f4934c927f178dd23dadc98f06a39d2caa890c2b348afeecd2929c7"} Nov 03 22:12:23.116613 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:23.116489 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" event={"ID":"bf8fa970-77b7-4f41-9a2d-a586343c51da","Type":"ContainerStarted","Data":"fd54fc1223b70cc26362825bcf3408f370a21fd68c93f4463c66538474cb6485"} Nov 03 22:12:23.116613 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:23.116576 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:23.117377 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:23.117332 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:23.195868 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:23.195788 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pjwnk" podStartSLOduration=1.489275977 podStartE2EDuration="9.195769767s" totalImagesPullingTime="7.70649379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:23.154594701 +0000 UTC m=+9.743650390" watchObservedRunningTime="2026-11-03 22:12:23.195769767 +0000 UTC m=+9.784825459" Nov 03 22:12:24.120985 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:24.120943 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" event={"ID":"bf8fa970-77b7-4f41-9a2d-a586343c51da","Type":"ContainerStarted","Data":"ecc79d87651a86ef7d4fbbeef174ad029509247481d1e817605c25f969cc2326"} Nov 03 22:12:24.124236 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:24.124209 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7b4rf" event={"ID":"677c391f-bc16-46c8-9439-02a1f3bf934d","Type":"ContainerStarted","Data":"fd596f0e87fc082eae3cc9fe358ee9a19a29daa712191d7e140bfb38f5bd48d8"} Nov 03 22:12:24.167108 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:24.166823 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" podStartSLOduration=1.951684201 podStartE2EDuration="10.166803347s" totalImagesPullingTime="8.215119146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:24.165803883 +0000 UTC m=+10.754859574" watchObservedRunningTime="2026-11-03 22:12:24.166803347 +0000 UTC m=+10.755859038" Nov 03 22:12:24.210106 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:24.210045 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7b4rf" podStartSLOduration=2.242276339 podStartE2EDuration="10.210027832s" totalImagesPullingTime="7.967751493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:24.209137237 +0000 UTC m=+10.798192927" watchObservedRunningTime="2026-11-03 22:12:24.210027832 +0000 UTC m=+10.799083523" Nov 03 22:12:24.944799 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:24.944762 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:24.944953 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:24.944803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:24.944953 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:24.944806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:24.944953 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:24.944917 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:24.945080 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:24.945016 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:24.945112 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:24.945094 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:25.127646 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:25.127621 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hmmwm" event={"ID":"805b7a3b-03f1-4799-a283-235314973feb","Type":"ContainerStarted","Data":"88e99a06068857c236421dee3a1f036fa7212a4bf13b48e5d20f7d4cb61be827"} Nov 03 22:12:25.129727 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:25.129700 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="d502b6d2b231ed514545a3b283416145b3f16447087fc75eae34cbec1682477c" exitCode=0 Nov 03 22:12:25.129817 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:25.129800 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" event={"ID":"2ac77e3c-256e-4a95-8106-de9bda8864c5","Type":"ContainerDied","Data":"d502b6d2b231ed514545a3b283416145b3f16447087fc75eae34cbec1682477c"} Nov 03 22:12:25.172367 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:25.172324 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hmmwm" podStartSLOduration=3.901688703 podStartE2EDuration="12.172306446s" totalImagesPullingTime="8.270617743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:25.171587338 +0000 UTC m=+11.760643028" watchObservedRunningTime="2026-11-03 22:12:25.172306446 +0000 UTC m=+11.761362129" Nov 03 22:12:26.134902 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:26.134822 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="221456ff2100528f163c05009c2a1671434bef4b0c051f7ea44ed1f6c0b71ddc" exitCode=0 Nov 03 22:12:26.135580 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:26.134911 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" event={"ID":"2ac77e3c-256e-4a95-8106-de9bda8864c5","Type":"ContainerDied","Data":"221456ff2100528f163c05009c2a1671434bef4b0c051f7ea44ed1f6c0b71ddc"} Nov 03 22:12:26.136561 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:26.136522 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" event={"ID":"8fc6de3b-56b8-4344-b7ab-3ad21f645803","Type":"ContainerStarted","Data":"cfb15552e759671683050b27a285e8150fe7e4c4176ed71fe20248606ed903da"} Nov 03 22:12:26.254256 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:26.254194 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" podStartSLOduration=2.365958675 podStartE2EDuration="12.254175149s" totalImagesPullingTime="9.888216474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:26.253556973 +0000 UTC m=+12.842612688" watchObservedRunningTime="2026-11-03 22:12:26.254175149 +0000 UTC m=+12.843230838" Nov 03 22:12:26.944622 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:26.944590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:26.944807 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:26.944587 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:26.944807 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:26.944701 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:26.944807 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:26.944716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:26.944915 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:26.944824 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:26.944915 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:26.944892 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:27.142026 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:27.141992 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="967e2abe1809e6ed13335f6c11ca3e0e5a0d6c7d1f31344b7ce3fb7c7929ba8e" exitCode=0 Nov 03 22:12:27.142642 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:27.142585 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" event={"ID":"2ac77e3c-256e-4a95-8106-de9bda8864c5","Type":"ContainerDied","Data":"967e2abe1809e6ed13335f6c11ca3e0e5a0d6c7d1f31344b7ce3fb7c7929ba8e"} Nov 03 22:12:28.944157 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:28.943955 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:28.944574 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:28.944052 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:28.944574 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:28.944315 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:28.944574 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:28.944065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:28.944574 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:28.944370 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:28.944574 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:28.944411 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:29.149882 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:29.149765 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="b36b9a12c1f54bd4c7b97442a1cba64f08b722a53bd7a1d6ad151b7caf30aa1b" exitCode=0 Nov 03 22:12:29.149882 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:29.149812 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" event={"ID":"2ac77e3c-256e-4a95-8106-de9bda8864c5","Type":"ContainerDied","Data":"b36b9a12c1f54bd4c7b97442a1cba64f08b722a53bd7a1d6ad151b7caf30aa1b"} Nov 03 22:12:30.155165 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.154930 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="479688a43b62224c324f432951ad096325f22fe1e00d98df3ffa359071585e73" exitCode=0 Nov 03 22:12:30.155565 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.155011 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" event={"ID":"2ac77e3c-256e-4a95-8106-de9bda8864c5","Type":"ContainerDied","Data":"479688a43b62224c324f432951ad096325f22fe1e00d98df3ffa359071585e73"} Nov 03 22:12:30.157971 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.157948 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:12:30.158390 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.158370 2580 generic.go:356] "Generic (PLEG): container finished" podID="44c8b7d7-71ad-47ee-9aad-be7f366974f5" containerID="82ec8f71575887f024b0480ab34d429de4998e60223323787289fc2068e51eab" exitCode=1 Nov 03 22:12:30.158469 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.158404 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerStarted","Data":"0ce48c862bbaaf6d41f8d84b04676fd36272bcbc3bae70e005f9292b6bef8986"} Nov 03 22:12:30.158469 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.158463 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerStarted","Data":"56db7ff626ce638a1f4ed90b05e4332e2215d2a72ba47e27f93ff01afd7919d1"} Nov 03 22:12:30.158529 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.158494 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerDied","Data":"82ec8f71575887f024b0480ab34d429de4998e60223323787289fc2068e51eab"} Nov 03 22:12:30.158557 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.158529 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerStarted","Data":"45db9e6a79646119c3516bc8d41ec70499e4908f1a5dba12a9143029c9584dd3"} Nov 03 22:12:30.639595 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.639503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:30.639595 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.639562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:30.639779 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.639643 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:30.639779 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.639659 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:30.639779 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.639693 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.639680055 +0000 UTC m=+33.228735723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 22:12:30.639779 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.639713 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret podName:05ab17d1-b5dc-4cb6-bf8e-eb650c724aef nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.639699852 +0000 UTC m=+33.228755521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret") pod "global-pull-secret-syncer-vk57x" (UID: "05ab17d1-b5dc-4cb6-bf8e-eb650c724aef") : object "kube-system"/"original-pull-secret" not registered Nov 03 22:12:30.740425 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.740390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:30.740655 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.740531 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 22:12:30.740655 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.740552 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 22:12:30.740655 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.740561 2580 projected.go:196] Error preparing data for projected volume kube-api-access-6sp7f for pod openshift-network-diagnostics/network-check-target-g7xxx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:30.740655 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.740608 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f podName:019bf19b-ca0e-4a02-af50-d6450151540e nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.740596719 +0000 UTC m=+33.329652387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6sp7f" (UniqueName: "kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f") pod "network-check-target-g7xxx" (UID: "019bf19b-ca0e-4a02-af50-d6450151540e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 22:12:30.944120 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.944091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:30.944120 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.944123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:30.944362 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:30.944090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:30.944362 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.944191 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:30.944362 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.944259 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:30.944362 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:30.944345 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:31.163513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:31.163481 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" event={"ID":"2ac77e3c-256e-4a95-8106-de9bda8864c5","Type":"ContainerStarted","Data":"a9fd9846420b1b603365ebe7785502f7accfef725a1caf86e8b9137019fef4c3"} Nov 03 22:12:31.166404 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:31.166384 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:12:31.166746 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:31.166722 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerStarted","Data":"6c427b5f73e532c8fa101f5f1a0939be8d0178d6e93a17ffa30293e68a34c551"} Nov 03 22:12:31.166804 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:31.166775 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerStarted","Data":"ed2dcb08773da101cfcfb2f7d87c1481e0537a52e45675fd982bfbb36ea557d6"} Nov 03 22:12:31.210835 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:31.210756 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nrm7n" podStartSLOduration=7.331523453 podStartE2EDuration="17.210744414s" totalImagesPullingTime="9.879220961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:31.210142112 +0000 UTC m=+17.799197804" watchObservedRunningTime="2026-11-03 22:12:31.210744414 +0000 UTC m=+17.799800103" Nov 03 22:12:32.943943 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:32.943909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:32.943943 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:32.943942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:32.944480 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:32.944028 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:32.944480 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:32.944053 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:32.944480 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:32.944091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:32.944480 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:32.944171 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:33.173580 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:33.173553 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:12:33.173917 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:33.173895 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerStarted","Data":"e348e7f47daa4d52878c75f1937cdbcd5709032d4f8efbb794bd922fc12d928b"} Nov 03 22:12:34.943904 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:34.943730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:34.944468 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:34.943788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:34.944468 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:34.943797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:34.944468 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:34.943991 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:34.944468 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:34.944100 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:34.944468 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:34.944167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:35.180898 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:35.180868 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:12:35.181247 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:35.181216 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerStarted","Data":"bfadfc4a631eff0fdb3cb6f5e36663aaeeab1ccb541b2d3c2f9fceb7a3d62123"} Nov 03 22:12:35.181501 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:35.181481 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:35.181637 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:35.181620 2580 scope.go:122] "RemoveContainer" containerID="82ec8f71575887f024b0480ab34d429de4998e60223323787289fc2068e51eab" Nov 03 22:12:35.197749 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:35.197701 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:36.188380 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.188353 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:12:36.188851 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.188767 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" event={"ID":"44c8b7d7-71ad-47ee-9aad-be7f366974f5","Type":"ContainerStarted","Data":"c164e1e3bce1e567607b21b1a5a2372d1765c86bc0ce89cb98c997c9ce375aaf"} Nov 03 22:12:36.188909 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.188892 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:12:36.189150 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.189131 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:36.203497 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.203466 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:36.281485 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.281426 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" podStartSLOduration=7.800969259 podStartE2EDuration="22.281409001s" totalImagesPullingTime="14.480439742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:36.280772891 +0000 UTC m=+22.869828580" watchObservedRunningTime="2026-11-03 22:12:36.281409001 +0000 UTC m=+22.870464692" Nov 03 22:12:36.523134 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.523065 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vk57x"] Nov 03 22:12:36.523279 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.523184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:36.523346 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:36.523304 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:36.525673 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.525649 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g7xxx"] Nov 03 22:12:36.525791 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.525741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:36.525831 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:36.525820 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:36.528607 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.528588 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b27xr"] Nov 03 22:12:36.528683 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:36.528675 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:36.528769 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:36.528753 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:37.191676 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:37.191645 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:12:37.944621 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:37.944586 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:37.944867 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:37.944749 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:37.944867 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:37.944809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:37.944939 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:37.944916 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:38.193915 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:38.193888 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:12:38.944213 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:38.944187 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:38.944426 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:38.944301 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:39.943730 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:39.943701 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:39.944224 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:39.943811 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:39.944224 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:39.943848 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:39.944224 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:39.943934 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:40.944161 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:40.944134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:40.944533 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:40.944237 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vk57x" podUID="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" Nov 03 22:12:41.944075 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:41.944035 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:41.944249 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:41.944040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:41.944249 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:41.944167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g7xxx" podUID="019bf19b-ca0e-4a02-af50-d6450151540e" Nov 03 22:12:41.944249 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:41.944221 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b27xr" podUID="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" Nov 03 22:12:42.681766 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.681737 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeReady" Nov 03 22:12:42.681935 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.681912 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 22:12:42.729958 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.729929 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d"] Nov 03 22:12:42.739468 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.739445 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l44xs"] Nov 03 22:12:42.739604 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.739570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.745835 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.745815 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-grdpd"] Nov 03 22:12:42.745955 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.745937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l44xs" Nov 03 22:12:42.750449 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.750426 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xcmlx"] Nov 03 22:12:42.750562 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.750524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-grdpd" Nov 03 22:12:42.753249 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.752711 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6"] Nov 03 22:12:42.753249 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.753024 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.755171 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.755156 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68496b87c-6xdvq"] Nov 03 22:12:42.755310 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.755276 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:42.757681 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.757666 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv"] Nov 03 22:12:42.757836 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.757810 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.760809 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.760788 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 22:12:42.760969 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.760947 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vn97d\"" Nov 03 22:12:42.761057 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.761005 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2"] Nov 03 22:12:42.761161 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.761146 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.770484 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.770460 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-t67lk"] Nov 03 22:12:42.770836 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.770580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:42.772678 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.772653 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 22:12:42.772772 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.772701 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.772772 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.772742 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.772873 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.772831 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.772964 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.772933 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.773922 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.773906 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-xsr82"] Nov 03 22:12:42.776043 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.774662 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:12:42.776043 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.774888 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.776043 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.775733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-t67lk" Nov 03 22:12:42.776326 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.776285 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.783239 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.783216 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2"] Nov 03 22:12:42.784320 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.783462 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:42.787795 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.786605 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zrq7h"] Nov 03 22:12:42.787795 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.787601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" Nov 03 22:12:42.789938 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.789915 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:12:42.792672 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.792648 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fd57577d7-9mscc"] Nov 03 22:12:42.793389 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.793325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:42.795955 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.795936 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-655957577d-lgbk6"] Nov 03 22:12:42.796643 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.796621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:42.801128 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.800536 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc"] Nov 03 22:12:42.801565 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.801547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:42.804371 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.804353 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8vfzp"] Nov 03 22:12:42.804490 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.804475 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" Nov 03 22:12:42.807685 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.807665 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vpk4w"] Nov 03 22:12:42.807797 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.807780 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:42.810156 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810137 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l44xs"] Nov 03 22:12:42.810260 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810175 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-grdpd"] Nov 03 22:12:42.810260 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810196 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d"] Nov 03 22:12:42.810260 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810221 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6"] Nov 03 22:12:42.810260 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810241 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-t67lk"] Nov 03 22:12:42.810260 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810260 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xcmlx"] Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810279 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2"] Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810410 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zrq7h"] Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810442 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc"] Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810462 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-655957577d-lgbk6"] Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810484 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2"] Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810509 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-xsr82"] Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810534 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68496b87c-6xdvq"] Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810554 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8vfzp"] Nov 03 22:12:42.810587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810578 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv"] Nov 03 22:12:42.811036 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810599 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vpk4w"] Nov 03 22:12:42.811036 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.810619 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fd57577d7-9mscc"] Nov 03 22:12:42.824137 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9qh\" (UniqueName: \"kubernetes.io/projected/60151974-97b1-4371-9831-762b096bc3fb-kube-api-access-bz9qh\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.824248 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.824248 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824160 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:42.824248 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-default-certificate\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.824248 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxxf\" (UniqueName: \"kubernetes.io/projected/0a18566a-0818-494c-989b-da2abea7edd9-kube-api-access-pvxxf\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.824248 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck4pl\" (UniqueName: \"kubernetes.io/projected/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-kube-api-access-ck4pl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.824248 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmw5v\" (UniqueName: \"kubernetes.io/projected/63867b60-8f1a-44a6-bbf1-8e521d232081-kube-api-access-vmw5v\") pod \"kas-connection-checker-685895c7c9-grdpd\" (UID: \"63867b60-8f1a-44a6-bbf1-8e521d232081\") " pod="kube-system/kas-connection-checker-685895c7c9-grdpd" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25gv\" (UniqueName: \"kubernetes.io/projected/375eaed5-066a-4615-bc60-09e343aec21b-kube-api-access-b25gv\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-stats-auth\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cqs8\" (UniqueName: \"kubernetes.io/projected/7c35b7b2-f209-4ff3-ab43-d8533fcdf341-kube-api-access-6cqs8\") pod \"kas-connection-checker-685895c7c9-t67lk\" (UID: \"7c35b7b2-f209-4ff3-ab43-d8533fcdf341\") " pod="kube-system/kas-connection-checker-685895c7c9-t67lk" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-tmp\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60151974-97b1-4371-9831-762b096bc3fb-serving-cert\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cf0d1c58-9465-423d-b479-50956f62c7f2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:42.824601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/375eaed5-066a-4615-bc60-09e343aec21b-config\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.825119 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824645 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/375eaed5-066a-4615-bc60-09e343aec21b-serving-cert\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.825119 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.825119 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/cf0d1c58-9465-423d-b479-50956f62c7f2-kube-api-access-54ggj\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:42.825119 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:42.825119 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62rmb\" (UniqueName: \"kubernetes.io/projected/a714258f-ba4e-4792-853b-7d20411d3ca7-kube-api-access-62rmb\") pod \"kas-connection-checker-685895c7c9-l44xs\" (UID: \"a714258f-ba4e-4792-853b-7d20411d3ca7\") " pod="kube-system/kas-connection-checker-685895c7c9-l44xs" Nov 03 22:12:42.825119 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-snapshots\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.825119 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.825119 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpjd\" (UniqueName: \"kubernetes.io/projected/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-kube-api-access-rzpjd\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:42.825119 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.824878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.864323 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.864283 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 22:12:42.864500 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.864308 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 22:12:42.864500 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.864332 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:12:42.864645 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.864547 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d79qx\"" Nov 03 22:12:42.864756 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.864740 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 22:12:42.864841 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.864796 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 22:12:42.865033 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865016 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 22:12:42.865033 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865026 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865033 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-g7tv5\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865049 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xf9z9\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865082 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865086 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l7j9d\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865092 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wwdkz\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865107 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865092 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-794rw\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865097 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-dg7lc\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865120 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wwhr4\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865128 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865133 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6vzq\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865146 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 22:12:42.865153 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865148 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h4rt7\"" Nov 03 22:12:42.865563 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865176 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:12:42.865563 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865202 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svbdj\"" Nov 03 22:12:42.865563 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865257 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7gr4r\"" Nov 03 22:12:42.865563 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865334 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:12:42.865563 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.865344 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:12:42.870178 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870159 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.870260 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870184 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.870431 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870418 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.870577 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870562 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 22:12:42.870651 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870613 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.870651 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870619 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.870747 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870720 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 22:12:42.870798 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870745 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 22:12:42.870798 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870780 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.870798 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870786 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 22:12:42.870973 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870795 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:12:42.870973 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870869 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.870973 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870911 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.870973 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870959 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.870973 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870963 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 22:12:42.871177 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870987 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.871177 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870999 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.871177 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.870964 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.871177 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.871022 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.871177 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.871031 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:12:42.871177 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.871017 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.871177 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.871056 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:12:42.871395 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.871304 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 22:12:42.879176 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.879156 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:12:42.879306 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.879278 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 22:12:42.926007 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.925977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4nt\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:42.926146 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:42.926146 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-snapshots\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.926224 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:42.926310 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.926379 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:42.926379 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926358 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-config\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:42.926577 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:42.926642 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926590 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-snapshots\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.926642 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:42.926801 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926702 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nwmt\" (UniqueName: \"kubernetes.io/projected/0bdc4254-64f5-4613-b007-85bc708b2d9b-kube-api-access-2nwmt\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:42.926801 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskmm\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:42.926801 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926782 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:42.926801 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:42.926987 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:42.926987 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.926987 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzpl5\" (UniqueName: \"kubernetes.io/projected/305d6b5c-15d1-4c7e-8b31-db2c71402cd3-kube-api-access-fzpl5\") pod \"volume-data-source-validator-58c9f5f46d-fgqk2\" (UID: \"305d6b5c-15d1-4c7e-8b31-db2c71402cd3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" Nov 03 22:12:42.926987 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60151974-97b1-4371-9831-762b096bc3fb-serving-cert\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.926987 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926951 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.926987 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.926961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cf0d1c58-9465-423d-b479-50956f62c7f2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:42.927257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-tmp-dir\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:42.927257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:42.927257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:42.927257 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:42.927533 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927216 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:42.927533 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:42.927533 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:42.927337 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:12:42.927533 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27xh4\" (UniqueName: \"kubernetes.io/projected/bcefadd9-c607-447d-85ce-fe31bc85e875-kube-api-access-27xh4\") pod \"network-check-source-6f78d59f99-q9cfc\" (UID: \"bcefadd9-c607-447d-85ce-fe31bc85e875\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" Nov 03 22:12:42.927533 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927416 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.927533 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:42.927425 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls podName:fdd2ca77-cb6a-4b10-906c-841c51aea0c6 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:43.427407008 +0000 UTC m=+30.016462698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hdwl2" (UID: "fdd2ca77-cb6a-4b10-906c-841c51aea0c6") : secret "samples-operator-tls" not found Nov 03 22:12:42.927533 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:42.927533 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpjd\" (UniqueName: \"kubernetes.io/projected/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-kube-api-access-rzpjd\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-serving-cert\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9qh\" (UniqueName: \"kubernetes.io/projected/60151974-97b1-4371-9831-762b096bc3fb-kube-api-access-bz9qh\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:42.927652 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:42.927700 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:43.42768356 +0000 UTC m=+30.016739233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : secret "router-metrics-certs-default" not found Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927720 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cf0d1c58-9465-423d-b479-50956f62c7f2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck4pl\" (UniqueName: \"kubernetes.io/projected/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-kube-api-access-ck4pl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-config-volume\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-trusted-ca\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25gv\" (UniqueName: \"kubernetes.io/projected/375eaed5-066a-4615-bc60-09e343aec21b-kube-api-access-b25gv\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.927861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927816 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-tmp\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:42.927952 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:43.427936884 +0000 UTC m=+30.016992577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : configmap references non-existent config key: service-ca.crt Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.927973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:42.927985 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:42.928032 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls podName:cf0d1c58-9465-423d-b479-50956f62c7f2 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:43.428020339 +0000 UTC m=+30.017076007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v9lq6" (UID: "cf0d1c58-9465-423d-b479-50956f62c7f2") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/375eaed5-066a-4615-bc60-09e343aec21b-config\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/375eaed5-066a-4615-bc60-09e343aec21b-serving-cert\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxxf\" (UniqueName: \"kubernetes.io/projected/0a18566a-0818-494c-989b-da2abea7edd9-kube-api-access-pvxxf\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bdc4254-64f5-4613-b007-85bc708b2d9b-cert\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928218 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-tmp\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62rmb\" (UniqueName: \"kubernetes.io/projected/a714258f-ba4e-4792-853b-7d20411d3ca7-kube-api-access-62rmb\") pod \"kas-connection-checker-685895c7c9-l44xs\" (UID: \"a714258f-ba4e-4792-853b-7d20411d3ca7\") " pod="kube-system/kas-connection-checker-685895c7c9-l44xs" Nov 03 22:12:42.928520 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmw5v\" (UniqueName: \"kubernetes.io/projected/63867b60-8f1a-44a6-bbf1-8e521d232081-kube-api-access-vmw5v\") pod \"kas-connection-checker-685895c7c9-grdpd\" (UID: \"63867b60-8f1a-44a6-bbf1-8e521d232081\") " pod="kube-system/kas-connection-checker-685895c7c9-grdpd" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-stats-auth\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cqs8\" (UniqueName: \"kubernetes.io/projected/7c35b7b2-f209-4ff3-ab43-d8533fcdf341-kube-api-access-6cqs8\") pod \"kas-connection-checker-685895c7c9-t67lk\" (UID: \"7c35b7b2-f209-4ff3-ab43-d8533fcdf341\") " pod="kube-system/kas-connection-checker-685895c7c9-t67lk" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/cf0d1c58-9465-423d-b479-50956f62c7f2-kube-api-access-54ggj\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-default-certificate\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928630 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/375eaed5-066a-4615-bc60-09e343aec21b-config\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx28g\" (UniqueName: \"kubernetes.io/projected/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-kube-api-access-cx28g\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xlx2\" (UniqueName: \"kubernetes.io/projected/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-kube-api-access-8xlx2\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:42.929185 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.928984 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.931211 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.931180 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/375eaed5-066a-4615-bc60-09e343aec21b-serving-cert\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.931319 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.931255 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.931380 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.931347 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60151974-97b1-4371-9831-762b096bc3fb-serving-cert\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.931454 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.931439 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-default-certificate\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.931549 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.931530 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-stats-auth\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.943995 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.943954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:42.952216 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.952199 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpjd\" (UniqueName: \"kubernetes.io/projected/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-kube-api-access-rzpjd\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:42.953151 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.953106 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62rmb\" (UniqueName: \"kubernetes.io/projected/a714258f-ba4e-4792-853b-7d20411d3ca7-kube-api-access-62rmb\") pod \"kas-connection-checker-685895c7c9-l44xs\" (UID: \"a714258f-ba4e-4792-853b-7d20411d3ca7\") " pod="kube-system/kas-connection-checker-685895c7c9-l44xs" Nov 03 22:12:42.954389 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.954333 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25gv\" (UniqueName: \"kubernetes.io/projected/375eaed5-066a-4615-bc60-09e343aec21b-kube-api-access-b25gv\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:42.954534 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.954508 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck4pl\" (UniqueName: \"kubernetes.io/projected/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-kube-api-access-ck4pl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:42.954678 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.954655 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxxf\" (UniqueName: \"kubernetes.io/projected/0a18566a-0818-494c-989b-da2abea7edd9-kube-api-access-pvxxf\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:42.954732 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.954698 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9qh\" (UniqueName: \"kubernetes.io/projected/60151974-97b1-4371-9831-762b096bc3fb-kube-api-access-bz9qh\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:42.954816 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.954796 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cqs8\" (UniqueName: \"kubernetes.io/projected/7c35b7b2-f209-4ff3-ab43-d8533fcdf341-kube-api-access-6cqs8\") pod \"kas-connection-checker-685895c7c9-t67lk\" (UID: \"7c35b7b2-f209-4ff3-ab43-d8533fcdf341\") " pod="kube-system/kas-connection-checker-685895c7c9-t67lk" Nov 03 22:12:42.954873 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.954819 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmw5v\" (UniqueName: \"kubernetes.io/projected/63867b60-8f1a-44a6-bbf1-8e521d232081-kube-api-access-vmw5v\") pod \"kas-connection-checker-685895c7c9-grdpd\" (UID: \"63867b60-8f1a-44a6-bbf1-8e521d232081\") " pod="kube-system/kas-connection-checker-685895c7c9-grdpd" Nov 03 22:12:42.960576 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.960558 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9wkj9\"" Nov 03 22:12:42.960763 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.960745 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/cf0d1c58-9465-423d-b479-50956f62c7f2-kube-api-access-54ggj\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:42.960821 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:42.960807 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:12:43.029727 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.029659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.029727 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.029720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx28g\" (UniqueName: \"kubernetes.io/projected/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-kube-api-access-cx28g\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:43.029933 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.029739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xlx2\" (UniqueName: \"kubernetes.io/projected/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-kube-api-access-8xlx2\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:43.029933 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.029757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.029933 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.029782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4nt\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.029933 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.029796 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:43.029933 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.029815 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655957577d-lgbk6: secret "image-registry-tls" not found Nov 03 22:12:43.029933 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.029798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.029933 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.029870 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:43.029933 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.029895 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd57577d7-9mscc: secret "image-registry-tls" not found Nov 03 22:12:43.030278 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.029872 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls podName:01e345d4-1cb8-4e4f-822d-d71b582b53b3 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:43.529853424 +0000 UTC m=+30.118909092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls") pod "image-registry-655957577d-lgbk6" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3") : secret "image-registry-tls" not found Nov 03 22:12:43.030278 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.029993 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls podName:d0dc65c0-bcea-4495-bf55-5095632b71b1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:43.529979326 +0000 UTC m=+30.119034997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls") pod "image-registry-5fd57577d7-9mscc" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1") : secret "image-registry-tls" not found Nov 03 22:12:43.030278 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.030278 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-config\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:43.030278 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.030278 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.030278 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwmt\" (UniqueName: \"kubernetes.io/projected/0bdc4254-64f5-4613-b007-85bc708b2d9b-kube-api-access-2nwmt\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:43.030278 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskmm\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.030278 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpl5\" (UniqueName: \"kubernetes.io/projected/305d6b5c-15d1-4c7e-8b31-db2c71402cd3-kube-api-access-fzpl5\") pod \"volume-data-source-validator-58c9f5f46d-fgqk2\" (UID: \"305d6b5c-15d1-4c7e-8b31-db2c71402cd3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-tmp-dir\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xh4\" (UniqueName: \"kubernetes.io/projected/bcefadd9-c607-447d-85ce-fe31bc85e875-kube-api-access-27xh4\") pod \"network-check-source-6f78d59f99-q9cfc\" (UID: \"bcefadd9-c607-447d-85ce-fe31bc85e875\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-serving-cert\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-config-volume\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:43.030805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-trusted-ca\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.030827 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030871 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.030885 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:12:43.530870649 +0000 UTC m=+30.119926331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030887 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-config\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.030756 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.031004 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.030988 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.031028 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.031084 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert podName:c9994b27-e2c7-4e4a-ab0c-796c19becb5c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:43.531069535 +0000 UTC m=+30.120125217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zrq7h" (UID: "c9994b27-e2c7-4e4a-ab0c-796c19becb5c") : secret "networking-console-plugin-cert" not found Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.031364 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.031481 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.031384 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.032171 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.031623 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-config-volume\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:43.032171 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.031914 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-trusted-ca\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:43.032582 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.032560 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-tmp-dir\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:43.032687 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.032667 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.032787 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.032767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bdc4254-64f5-4613-b007-85bc708b2d9b-cert\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:43.033109 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.033072 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.033809 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.033779 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.033967 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.033949 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.034144 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.034127 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-serving-cert\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:43.034404 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.034387 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.034907 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.034890 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bdc4254-64f5-4613-b007-85bc708b2d9b-cert\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:43.051615 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.051598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:43.054140 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.054083 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx28g\" (UniqueName: \"kubernetes.io/projected/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-kube-api-access-cx28g\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:43.054815 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.054793 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xlx2\" (UniqueName: \"kubernetes.io/projected/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-kube-api-access-8xlx2\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:43.054926 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.054903 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskmm\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.055073 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.055052 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwmt\" (UniqueName: \"kubernetes.io/projected/0bdc4254-64f5-4613-b007-85bc708b2d9b-kube-api-access-2nwmt\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:43.055179 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.055163 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4nt\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.055284 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.055264 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.055344 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.055274 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.055729 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.055708 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xh4\" (UniqueName: \"kubernetes.io/projected/bcefadd9-c607-447d-85ce-fe31bc85e875-kube-api-access-27xh4\") pod \"network-check-source-6f78d59f99-q9cfc\" (UID: \"bcefadd9-c607-447d-85ce-fe31bc85e875\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" Nov 03 22:12:43.056432 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.056415 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpl5\" (UniqueName: \"kubernetes.io/projected/305d6b5c-15d1-4c7e-8b31-db2c71402cd3-kube-api-access-fzpl5\") pod \"volume-data-source-validator-58c9f5f46d-fgqk2\" (UID: \"305d6b5c-15d1-4c7e-8b31-db2c71402cd3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" Nov 03 22:12:43.061199 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.061184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-l44xs" Nov 03 22:12:43.084212 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.084122 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-grdpd" Nov 03 22:12:43.092222 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.092197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:43.119012 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.118979 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:43.132457 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.132422 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-t67lk" Nov 03 22:12:43.138170 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.138141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:43.145106 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.145056 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" Nov 03 22:12:43.174904 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.174706 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" Nov 03 22:12:43.209322 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.208492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:43.419563 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.419538 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-grdpd"] Nov 03 22:12:43.420401 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:43.420319 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63867b60_8f1a_44a6_bbf1_8e521d232081.slice/crio-6fec7e3cb2ca04b9eb72c015cddf8c6d390ba71d13dd7e74b83861dad88e8482 WatchSource:0}: Error finding container 6fec7e3cb2ca04b9eb72c015cddf8c6d390ba71d13dd7e74b83861dad88e8482: Status 404 returned error can't find the container with id 6fec7e3cb2ca04b9eb72c015cddf8c6d390ba71d13dd7e74b83861dad88e8482 Nov 03 22:12:43.422108 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.422085 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-l44xs"] Nov 03 22:12:43.422171 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:43.422145 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda714258f_ba4e_4792_853b_7d20411d3ca7.slice/crio-7c6a86ac3f8b52977e833b5ad355764cfd791740e99e210c54aff67296ebb2ca WatchSource:0}: Error finding container 7c6a86ac3f8b52977e833b5ad355764cfd791740e99e210c54aff67296ebb2ca: Status 404 returned error can't find the container with id 7c6a86ac3f8b52977e833b5ad355764cfd791740e99e210c54aff67296ebb2ca Nov 03 22:12:43.432205 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.432183 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d"] Nov 03 22:12:43.432787 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:43.432762 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375eaed5_066a_4615_bc60_09e343aec21b.slice/crio-d83bfeb13bebe06dc72d8cd05482d56d69b5c83ab4f92008ac8962ba1f1bbb70 WatchSource:0}: Error finding container d83bfeb13bebe06dc72d8cd05482d56d69b5c83ab4f92008ac8962ba1f1bbb70: Status 404 returned error can't find the container with id d83bfeb13bebe06dc72d8cd05482d56d69b5c83ab4f92008ac8962ba1f1bbb70 Nov 03 22:12:43.434804 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.434757 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-xcmlx"] Nov 03 22:12:43.437598 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:43.437580 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60151974_97b1_4371_9831_762b096bc3fb.slice/crio-c8820d4b769ead6d669adc81d3580ddb6591a9077473b7dc56ee136349bd8461 WatchSource:0}: Error finding container c8820d4b769ead6d669adc81d3580ddb6591a9077473b7dc56ee136349bd8461: Status 404 returned error can't find the container with id c8820d4b769ead6d669adc81d3580ddb6591a9077473b7dc56ee136349bd8461 Nov 03 22:12:43.445005 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.444979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:43.445149 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.445133 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:43.445198 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.445187 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:12:43.445247 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.445130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:43.445247 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.445191 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls podName:cf0d1c58-9465-423d-b479-50956f62c7f2 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:44.445173944 +0000 UTC m=+31.034229632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v9lq6" (UID: "cf0d1c58-9465-423d-b479-50956f62c7f2") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:43.445247 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.445235 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls podName:fdd2ca77-cb6a-4b10-906c-841c51aea0c6 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:44.445224621 +0000 UTC m=+31.034280290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hdwl2" (UID: "fdd2ca77-cb6a-4b10-906c-841c51aea0c6") : secret "samples-operator-tls" not found Nov 03 22:12:43.445407 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.445308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:43.445407 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.445347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:43.445484 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.445425 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:44.445414159 +0000 UTC m=+31.034469826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : configmap references non-existent config key: service-ca.crt Nov 03 22:12:43.445484 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.445462 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:12:43.445573 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.445501 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:44.445491316 +0000 UTC m=+31.034546999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : secret "router-metrics-certs-default" not found Nov 03 22:12:43.505906 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.500626 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv"] Nov 03 22:12:43.505906 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.505863 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-xsr82"] Nov 03 22:12:43.527375 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.527274 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2"] Nov 03 22:12:43.529855 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:43.529828 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305d6b5c_15d1_4c7e_8b31_db2c71402cd3.slice/crio-93e9e80e1ff802ecac50970c92c0dcc41bfc164f371eefafc5fce32c07a913ad WatchSource:0}: Error finding container 93e9e80e1ff802ecac50970c92c0dcc41bfc164f371eefafc5fce32c07a913ad: Status 404 returned error can't find the container with id 93e9e80e1ff802ecac50970c92c0dcc41bfc164f371eefafc5fce32c07a913ad Nov 03 22:12:43.530098 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.530080 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-t67lk"] Nov 03 22:12:43.531071 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:43.531047 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c35b7b2_f209_4ff3_ab43_d8533fcdf341.slice/crio-5817d8adba03a158b2d68586b18d2985fa0a10ac1ddcf217e92ebf9f2d5fb732 WatchSource:0}: Error finding container 5817d8adba03a158b2d68586b18d2985fa0a10ac1ddcf217e92ebf9f2d5fb732: Status 404 returned error can't find the container with id 5817d8adba03a158b2d68586b18d2985fa0a10ac1ddcf217e92ebf9f2d5fb732 Nov 03 22:12:43.545877 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.545855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:43.545984 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.545934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:43.546042 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.546003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:43.546042 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546034 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:43.546180 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546049 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd57577d7-9mscc: secret "image-registry-tls" not found Nov 03 22:12:43.546180 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546089 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls podName:d0dc65c0-bcea-4495-bf55-5095632b71b1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:44.546075581 +0000 UTC m=+31.135131278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls") pod "image-registry-5fd57577d7-9mscc" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1") : secret "image-registry-tls" not found Nov 03 22:12:43.546180 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.546130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:43.546355 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546175 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:43.546355 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546241 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:12:44.54622559 +0000 UTC m=+31.135281275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:12:43.546355 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546267 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:43.546355 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546277 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655957577d-lgbk6: secret "image-registry-tls" not found Nov 03 22:12:43.546355 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546330 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls podName:01e345d4-1cb8-4e4f-822d-d71b582b53b3 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:44.546321115 +0000 UTC m=+31.135376796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls") pod "image-registry-655957577d-lgbk6" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3") : secret "image-registry-tls" not found Nov 03 22:12:43.546355 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546328 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:12:43.556478 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:43.546377 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert podName:c9994b27-e2c7-4e4a-ab0c-796c19becb5c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:44.546364641 +0000 UTC m=+31.135420309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zrq7h" (UID: "c9994b27-e2c7-4e4a-ab0c-796c19becb5c") : secret "networking-console-plugin-cert" not found Nov 03 22:12:43.556478 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.553596 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vpk4w"] Nov 03 22:12:43.556478 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:43.554375 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdc4254_64f5_4613_b007_85bc708b2d9b.slice/crio-459aa21bbbcdd85abcc8468397f971c9543d8e8f18ad39baafd7a2a5d87b2a44 WatchSource:0}: Error finding container 459aa21bbbcdd85abcc8468397f971c9543d8e8f18ad39baafd7a2a5d87b2a44: Status 404 returned error can't find the container with id 459aa21bbbcdd85abcc8468397f971c9543d8e8f18ad39baafd7a2a5d87b2a44 Nov 03 22:12:43.560028 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.559920 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc"] Nov 03 22:12:43.560193 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:43.560169 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcefadd9_c607_447d_85ce_fe31bc85e875.slice/crio-4bfc8057990bcef4eca76b5c95505191cbbd57978cd478ab8bd9b1006d77926f WatchSource:0}: Error finding container 4bfc8057990bcef4eca76b5c95505191cbbd57978cd478ab8bd9b1006d77926f: Status 404 returned error can't find the container with id 4bfc8057990bcef4eca76b5c95505191cbbd57978cd478ab8bd9b1006d77926f Nov 03 22:12:43.953156 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.953122 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:43.953629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.953617 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:43.969527 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.969501 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:12:43.969750 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.969553 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zdncz\"" Nov 03 22:12:43.969919 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:43.969645 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mq5fv\"" Nov 03 22:12:44.220000 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.219869 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vpk4w" event={"ID":"0bdc4254-64f5-4613-b007-85bc708b2d9b","Type":"ContainerStarted","Data":"459aa21bbbcdd85abcc8468397f971c9543d8e8f18ad39baafd7a2a5d87b2a44"} Nov 03 22:12:44.223554 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.223470 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" event={"ID":"305d6b5c-15d1-4c7e-8b31-db2c71402cd3","Type":"ContainerStarted","Data":"93e9e80e1ff802ecac50970c92c0dcc41bfc164f371eefafc5fce32c07a913ad"} Nov 03 22:12:44.229403 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.229353 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xcmlx" event={"ID":"60151974-97b1-4371-9831-762b096bc3fb","Type":"ContainerStarted","Data":"c8820d4b769ead6d669adc81d3580ddb6591a9077473b7dc56ee136349bd8461"} Nov 03 22:12:44.231406 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.231326 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" event={"ID":"33d39b02-7a8c-4edc-ae53-3ebd1e494f58","Type":"ContainerStarted","Data":"8bc09fba35c7598c9b4fe4cfe87d932b6b13d539014543558f0e99256cd5a770"} Nov 03 22:12:44.234963 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.234195 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l44xs" event={"ID":"a714258f-ba4e-4792-853b-7d20411d3ca7","Type":"ContainerStarted","Data":"6cb65e96009e5c6b330d1fa175714b6309de25e98298bfa9d113713ba0390c5f"} Nov 03 22:12:44.234963 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.234232 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-l44xs" event={"ID":"a714258f-ba4e-4792-853b-7d20411d3ca7","Type":"ContainerStarted","Data":"7c6a86ac3f8b52977e833b5ad355764cfd791740e99e210c54aff67296ebb2ca"} Nov 03 22:12:44.237203 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.237132 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" event={"ID":"375eaed5-066a-4615-bc60-09e343aec21b","Type":"ContainerStarted","Data":"d83bfeb13bebe06dc72d8cd05482d56d69b5c83ab4f92008ac8962ba1f1bbb70"} Nov 03 22:12:44.238790 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.238766 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" event={"ID":"bcefadd9-c607-447d-85ce-fe31bc85e875","Type":"ContainerStarted","Data":"4bfc8057990bcef4eca76b5c95505191cbbd57978cd478ab8bd9b1006d77926f"} Nov 03 22:12:44.240513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.240492 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" event={"ID":"0c3d06f5-c335-4cf2-a901-cae1f6c2e142","Type":"ContainerStarted","Data":"c9ddb67e3dd68da22f612d7158c8e0b43ad1b0970792439fa6e694b53a2256b3"} Nov 03 22:12:44.242228 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.242198 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-t67lk" event={"ID":"7c35b7b2-f209-4ff3-ab43-d8533fcdf341","Type":"ContainerStarted","Data":"97e3727939847ca9b846316f81693bfc4fa6bcfa4617067869c3bfcfc0337303"} Nov 03 22:12:44.242335 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.242238 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-t67lk" event={"ID":"7c35b7b2-f209-4ff3-ab43-d8533fcdf341","Type":"ContainerStarted","Data":"5817d8adba03a158b2d68586b18d2985fa0a10ac1ddcf217e92ebf9f2d5fb732"} Nov 03 22:12:44.244388 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.244364 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-grdpd" event={"ID":"63867b60-8f1a-44a6-bbf1-8e521d232081","Type":"ContainerStarted","Data":"0cff65a5a4ecb99a4c8afab510804d4395ccc420d9280b545dd437b9bcd5fde2"} Nov 03 22:12:44.244492 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.244401 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-grdpd" event={"ID":"63867b60-8f1a-44a6-bbf1-8e521d232081","Type":"ContainerStarted","Data":"6fec7e3cb2ca04b9eb72c015cddf8c6d390ba71d13dd7e74b83861dad88e8482"} Nov 03 22:12:44.264664 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.264639 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:44.264889 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.264874 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:12:44.276886 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.276841 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-l44xs" podStartSLOduration=414.276829766 podStartE2EDuration="6m54.276829766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:44.275563151 +0000 UTC m=+30.864618842" watchObservedRunningTime="2026-11-03 22:12:44.276829766 +0000 UTC m=+30.865885453" Nov 03 22:12:44.282156 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.281808 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:44.368807 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.368749 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-t67lk" podStartSLOduration=414.368732286 podStartE2EDuration="6m54.368732286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:44.368179758 +0000 UTC m=+30.957235449" watchObservedRunningTime="2026-11-03 22:12:44.368732286 +0000 UTC m=+30.957787979" Nov 03 22:12:44.369193 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.369168 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-grdpd" podStartSLOduration=414.369158707 podStartE2EDuration="6m54.369158707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:44.320450487 +0000 UTC m=+30.909506179" watchObservedRunningTime="2026-11-03 22:12:44.369158707 +0000 UTC m=+30.958214400" Nov 03 22:12:44.456514 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.456477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:44.456695 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.456553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:44.456695 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.456586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:44.456695 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.456613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:44.456912 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.456835 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:44.456912 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.456909 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls podName:cf0d1c58-9465-423d-b479-50956f62c7f2 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.45689105 +0000 UTC m=+33.045946737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v9lq6" (UID: "cf0d1c58-9465-423d-b479-50956f62c7f2") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:44.457348 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.457326 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:12:44.457452 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.457383 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls podName:fdd2ca77-cb6a-4b10-906c-841c51aea0c6 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.457367402 +0000 UTC m=+33.046423085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hdwl2" (UID: "fdd2ca77-cb6a-4b10-906c-841c51aea0c6") : secret "samples-operator-tls" not found Nov 03 22:12:44.457452 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.457440 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:12:44.457555 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.457471 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.457460929 +0000 UTC m=+33.046516611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : secret "router-metrics-certs-default" not found Nov 03 22:12:44.457555 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.457533 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.457523465 +0000 UTC m=+33.046579147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : configmap references non-existent config key: service-ca.crt Nov 03 22:12:44.557358 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.557269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:44.557513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.557397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:44.557513 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.557449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:44.557613 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:44.557514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:44.557674 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.557655 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:12:44.557728 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.557724 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert podName:c9994b27-e2c7-4e4a-ab0c-796c19becb5c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.557705835 +0000 UTC m=+33.146761508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zrq7h" (UID: "c9994b27-e2c7-4e4a-ab0c-796c19becb5c") : secret "networking-console-plugin-cert" not found Nov 03 22:12:44.558114 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.558093 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:44.558201 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.558145 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.558131286 +0000 UTC m=+33.147186957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:12:44.558259 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.558207 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:44.558259 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.558217 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655957577d-lgbk6: secret "image-registry-tls" not found Nov 03 22:12:44.558259 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.558247 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls podName:01e345d4-1cb8-4e4f-822d-d71b582b53b3 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.558237301 +0000 UTC m=+33.147292971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls") pod "image-registry-655957577d-lgbk6" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3") : secret "image-registry-tls" not found Nov 03 22:12:44.558442 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.558315 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:44.558442 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.558325 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd57577d7-9mscc: secret "image-registry-tls" not found Nov 03 22:12:44.558442 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:44.558352 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls podName:d0dc65c0-bcea-4495-bf55-5095632b71b1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:46.558343181 +0000 UTC m=+33.147398854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls") pod "image-registry-5fd57577d7-9mscc" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1") : secret "image-registry-tls" not found Nov 03 22:12:46.257323 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.256882 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" event={"ID":"305d6b5c-15d1-4c7e-8b31-db2c71402cd3","Type":"ContainerStarted","Data":"f138e127e6f5b411ab64a71d1729061f8974d9c0811ad07d6d1b3bcad2caf906"} Nov 03 22:12:46.300647 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.300588 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" podStartSLOduration=375.039331964 podStartE2EDuration="6m17.300574065s" totalImagesPullingTime="2.261242101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:46.29899162 +0000 UTC m=+32.888047315" watchObservedRunningTime="2026-11-03 22:12:46.300574065 +0000 UTC m=+32.889629755" Nov 03 22:12:46.480748 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.480662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:46.481003 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.480750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:46.481003 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.480793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:46.481003 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.480830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:46.481133 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.481058 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:46.481133 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.481126 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls podName:cf0d1c58-9465-423d-b479-50956f62c7f2 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.481108095 +0000 UTC m=+37.070163769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v9lq6" (UID: "cf0d1c58-9465-423d-b479-50956f62c7f2") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:46.481569 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.481541 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:12:46.481688 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.481613 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls podName:fdd2ca77-cb6a-4b10-906c-841c51aea0c6 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.481594905 +0000 UTC m=+37.070650599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hdwl2" (UID: "fdd2ca77-cb6a-4b10-906c-841c51aea0c6") : secret "samples-operator-tls" not found Nov 03 22:12:46.481688 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.481677 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:12:46.481798 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.481704 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.481697095 +0000 UTC m=+37.070752762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : secret "router-metrics-certs-default" not found Nov 03 22:12:46.481798 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.481755 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.481749235 +0000 UTC m=+37.070804902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : configmap references non-existent config key: service-ca.crt Nov 03 22:12:46.582116 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.582056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:46.582282 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.582131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:46.582282 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.582201 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:46.582282 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.582262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:46.582440 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.582430 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:46.582496 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.582490 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.582473083 +0000 UTC m=+37.171528770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:12:46.582897 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.582868 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:46.582897 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.582891 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655957577d-lgbk6: secret "image-registry-tls" not found Nov 03 22:12:46.583061 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.582930 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls podName:01e345d4-1cb8-4e4f-822d-d71b582b53b3 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.582915861 +0000 UTC m=+37.171971533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls") pod "image-registry-655957577d-lgbk6" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3") : secret "image-registry-tls" not found Nov 03 22:12:46.583061 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.582968 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:46.583061 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.582975 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd57577d7-9mscc: secret "image-registry-tls" not found Nov 03 22:12:46.583061 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.583001 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls podName:d0dc65c0-bcea-4495-bf55-5095632b71b1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.582991021 +0000 UTC m=+37.172046692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls") pod "image-registry-5fd57577d7-9mscc" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1") : secret "image-registry-tls" not found Nov 03 22:12:46.583061 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.583046 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:12:46.583061 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.583068 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert podName:c9994b27-e2c7-4e4a-ab0c-796c19becb5c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.583061888 +0000 UTC m=+37.172117557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zrq7h" (UID: "c9994b27-e2c7-4e4a-ab0c-796c19becb5c") : secret "networking-console-plugin-cert" not found Nov 03 22:12:46.683601 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.683561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:46.683792 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.683640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:46.683854 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.683807 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:12:46.683907 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:46.683857 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:18.683844462 +0000 UTC m=+65.272900130 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : secret "metrics-daemon-secret" not found Nov 03 22:12:46.687762 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.687710 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:46.785366 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.785112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:46.796015 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.795984 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:46.902455 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:46.902415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:47.056553 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:47.056481 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:47.268921 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:47.268836 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" event={"ID":"0c3d06f5-c335-4cf2-a901-cae1f6c2e142","Type":"ContainerStarted","Data":"fdebe1a9c51c094574f6f1318dcf0b21dc295386164e4a8157af89e52a2725ed"} Nov 03 22:12:47.268921 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:47.268889 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:47.270345 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:47.270060 2580 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-xsr82 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Nov 03 22:12:47.270345 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:47.270116 2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" podUID="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Nov 03 22:12:47.316504 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:47.316415 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" podStartSLOduration=376.761661594 podStartE2EDuration="6m20.316402469s" totalImagesPullingTime="3.554740875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:47.314850431 +0000 UTC m=+33.903906122" watchObservedRunningTime="2026-11-03 22:12:47.316402469 +0000 UTC m=+33.905458156" Nov 03 22:12:47.374475 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:47.374442 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vk57x"] Nov 03 22:12:47.375720 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:47.375619 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ab17d1_b5dc_4cb6_bf8e_eb650c724aef.slice/crio-f8cd69b5fc7b0847591e23f59b2422bd6a7cf573e8f845423eb620d34750e09b WatchSource:0}: Error finding container f8cd69b5fc7b0847591e23f59b2422bd6a7cf573e8f845423eb620d34750e09b: Status 404 returned error can't find the container with id f8cd69b5fc7b0847591e23f59b2422bd6a7cf573e8f845423eb620d34750e09b Nov 03 22:12:47.416370 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:47.416333 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g7xxx"] Nov 03 22:12:47.436084 ip-10-0-133-193 kubenswrapper[2580]: W1103 22:12:47.436052 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019bf19b_ca0e_4a02_af50_d6450151540e.slice/crio-db7c2c0c5dee19fe354cad7fc4a0db4719d8ac45ad8aa898643221673a90abf6 WatchSource:0}: Error finding container db7c2c0c5dee19fe354cad7fc4a0db4719d8ac45ad8aa898643221673a90abf6: Status 404 returned error can't find the container with id db7c2c0c5dee19fe354cad7fc4a0db4719d8ac45ad8aa898643221673a90abf6 Nov 03 22:12:48.273477 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.273437 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xcmlx" event={"ID":"60151974-97b1-4371-9831-762b096bc3fb","Type":"ContainerStarted","Data":"d730e0fea10787854789487fd884518ace03e18e449353e4030f0fb28ac10108"} Nov 03 22:12:48.274880 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.274836 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vk57x" event={"ID":"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef","Type":"ContainerStarted","Data":"f8cd69b5fc7b0847591e23f59b2422bd6a7cf573e8f845423eb620d34750e09b"} Nov 03 22:12:48.276590 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.276555 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" event={"ID":"375eaed5-066a-4615-bc60-09e343aec21b","Type":"ContainerStarted","Data":"8cf712c7d94ef9c861c3a00b3a07c35a25103e7fd76bcaa8c3d9b7a4077c757b"} Nov 03 22:12:48.277950 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.277923 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g7xxx" event={"ID":"019bf19b-ca0e-4a02-af50-d6450151540e","Type":"ContainerStarted","Data":"db7c2c0c5dee19fe354cad7fc4a0db4719d8ac45ad8aa898643221673a90abf6"} Nov 03 22:12:48.279677 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.279655 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/0.log" Nov 03 22:12:48.279780 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.279696 2580 generic.go:356] "Generic (PLEG): container finished" podID="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" containerID="fdebe1a9c51c094574f6f1318dcf0b21dc295386164e4a8157af89e52a2725ed" exitCode=255 Nov 03 22:12:48.279780 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.279727 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" event={"ID":"0c3d06f5-c335-4cf2-a901-cae1f6c2e142","Type":"ContainerDied","Data":"fdebe1a9c51c094574f6f1318dcf0b21dc295386164e4a8157af89e52a2725ed"} Nov 03 22:12:48.280004 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.279989 2580 scope.go:122] "RemoveContainer" containerID="fdebe1a9c51c094574f6f1318dcf0b21dc295386164e4a8157af89e52a2725ed" Nov 03 22:12:48.317959 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.316712 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-xcmlx" podStartSLOduration=376.868181418 podStartE2EDuration="6m21.316693326s" totalImagesPullingTime="4.448511908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:48.315753687 +0000 UTC m=+34.904809382" watchObservedRunningTime="2026-11-03 22:12:48.316693326 +0000 UTC m=+34.905749016" Nov 03 22:12:48.360742 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.360086 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" podStartSLOduration=376.832488198 podStartE2EDuration="6m21.360065475s" totalImagesPullingTime="4.527577277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:48.358931648 +0000 UTC m=+34.947987339" watchObservedRunningTime="2026-11-03 22:12:48.360065475 +0000 UTC m=+34.949121165" Nov 03 22:12:48.690745 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.689265 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7vs67"] Nov 03 22:12:48.707100 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.707073 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.715866 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.715529 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vs67"] Nov 03 22:12:48.734805 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.734681 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zwzw5\"" Nov 03 22:12:48.807649 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.807574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5223b025-80ca-4412-99fb-eed48e128397-data-volume\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.807649 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.807637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5223b025-80ca-4412-99fb-eed48e128397-crio-socket\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.807861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.807727 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.807861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.807751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqjf\" (UniqueName: \"kubernetes.io/projected/5223b025-80ca-4412-99fb-eed48e128397-kube-api-access-shqjf\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.807861 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.807789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.825960 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.825906 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:12:48.826886 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.826111 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:12:48.908910 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.908871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5223b025-80ca-4412-99fb-eed48e128397-crio-socket\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.909114 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.908991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.909114 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.909019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqjf\" (UniqueName: \"kubernetes.io/projected/5223b025-80ca-4412-99fb-eed48e128397-kube-api-access-shqjf\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.909114 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.909063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.909254 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.909164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5223b025-80ca-4412-99fb-eed48e128397-data-volume\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.909545 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.909518 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5223b025-80ca-4412-99fb-eed48e128397-data-volume\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.909629 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.909598 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5223b025-80ca-4412-99fb-eed48e128397-crio-socket\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:48.909667 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:48.909655 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:12:48.909837 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:48.909713 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm podName:5223b025-80ca-4412-99fb-eed48e128397 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:49.409694153 +0000 UTC m=+35.998749828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vs67" (UID: "5223b025-80ca-4412-99fb-eed48e128397") : configmap "kube-rbac-proxy" not found Nov 03 22:12:48.910172 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:48.910156 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:12:48.910219 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:48.910208 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls podName:5223b025-80ca-4412-99fb-eed48e128397 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:49.410194135 +0000 UTC m=+35.999249808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vs67" (UID: "5223b025-80ca-4412-99fb-eed48e128397") : secret "insights-runtime-extractor-tls" not found Nov 03 22:12:48.939980 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:48.939953 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqjf\" (UniqueName: \"kubernetes.io/projected/5223b025-80ca-4412-99fb-eed48e128397-kube-api-access-shqjf\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:49.289656 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.289534 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" event={"ID":"33d39b02-7a8c-4edc-ae53-3ebd1e494f58","Type":"ContainerStarted","Data":"f99a44ce15ffde30f03e4d1f3f313f398c224127e352d5465a0ca122b827c935"} Nov 03 22:12:49.294215 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.294156 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" event={"ID":"bcefadd9-c607-447d-85ce-fe31bc85e875","Type":"ContainerStarted","Data":"590c6341a10358da781b7bf33faee4291aaa4d9fbe11e0a9bd23cc11e2874bca"} Nov 03 22:12:49.296392 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.296311 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:12:49.297085 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.297064 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/0.log" Nov 03 22:12:49.297182 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.297104 2580 generic.go:356] "Generic (PLEG): container finished" podID="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" containerID="0a268de110e22153c69a83cfaf9d6fed6ce349dd349ecd27279f24377dc95850" exitCode=255 Nov 03 22:12:49.297355 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.297270 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" event={"ID":"0c3d06f5-c335-4cf2-a901-cae1f6c2e142","Type":"ContainerDied","Data":"0a268de110e22153c69a83cfaf9d6fed6ce349dd349ecd27279f24377dc95850"} Nov 03 22:12:49.297587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.297469 2580 scope.go:122] "RemoveContainer" containerID="fdebe1a9c51c094574f6f1318dcf0b21dc295386164e4a8157af89e52a2725ed" Nov 03 22:12:49.297587 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.297502 2580 scope.go:122] "RemoveContainer" containerID="0a268de110e22153c69a83cfaf9d6fed6ce349dd349ecd27279f24377dc95850" Nov 03 22:12:49.297731 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:49.297705 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xsr82_openshift-console-operator(0c3d06f5-c335-4cf2-a901-cae1f6c2e142)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" podUID="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" Nov 03 22:12:49.300657 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.300542 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vpk4w" event={"ID":"0bdc4254-64f5-4613-b007-85bc708b2d9b","Type":"ContainerStarted","Data":"a65f9653110d53ef99f3f85fd1242ed2238ec216ed0b2c457d08a7136d1e1d47"} Nov 03 22:12:49.340010 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.339054 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" podStartSLOduration=377.35333749 podStartE2EDuration="6m22.339036237s" totalImagesPullingTime="4.985698747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:49.336829433 +0000 UTC m=+35.925885140" watchObservedRunningTime="2026-11-03 22:12:49.339036237 +0000 UTC m=+35.928091928" Nov 03 22:12:49.416326 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.416091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:49.416326 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.416167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:49.418349 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:49.416705 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:12:49.418349 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:49.416763 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls podName:5223b025-80ca-4412-99fb-eed48e128397 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.416747111 +0000 UTC m=+37.005802787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vs67" (UID: "5223b025-80ca-4412-99fb-eed48e128397") : secret "insights-runtime-extractor-tls" not found Nov 03 22:12:49.418828 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:49.418582 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:12:49.418828 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:49.418624 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm podName:5223b025-80ca-4412-99fb-eed48e128397 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:50.418612499 +0000 UTC m=+37.007668168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vs67" (UID: "5223b025-80ca-4412-99fb-eed48e128397") : configmap "kube-rbac-proxy" not found Nov 03 22:12:49.435754 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.435040 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vpk4w" podStartSLOduration=2.475354389 podStartE2EDuration="7.435022062s" totalImagesPullingTime="4.959667673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:49.433781658 +0000 UTC m=+36.022837349" watchObservedRunningTime="2026-11-03 22:12:49.435022062 +0000 UTC m=+36.024077753" Nov 03 22:12:49.483912 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:49.483796 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" podStartSLOduration=373.121061181 podStartE2EDuration="6m18.483779042s" totalImagesPullingTime="5.362717861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:49.483626279 +0000 UTC m=+36.072681968" watchObservedRunningTime="2026-11-03 22:12:49.483779042 +0000 UTC m=+36.072834730" Nov 03 22:12:50.306538 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.306502 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:12:50.306930 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.306914 2580 scope.go:122] "RemoveContainer" containerID="0a268de110e22153c69a83cfaf9d6fed6ce349dd349ecd27279f24377dc95850" Nov 03 22:12:50.307163 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.307137 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xsr82_openshift-console-operator(0c3d06f5-c335-4cf2-a901-cae1f6c2e142)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" podUID="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" Nov 03 22:12:50.429926 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.429883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:50.430096 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.429937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:50.430096 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.430006 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:12:50.430096 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.430073 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm podName:5223b025-80ca-4412-99fb-eed48e128397 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:52.430053595 +0000 UTC m=+39.019109271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vs67" (UID: "5223b025-80ca-4412-99fb-eed48e128397") : configmap "kube-rbac-proxy" not found Nov 03 22:12:50.430266 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.430148 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:12:50.430266 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.430191 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls podName:5223b025-80ca-4412-99fb-eed48e128397 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:52.430176575 +0000 UTC m=+39.019232260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vs67" (UID: "5223b025-80ca-4412-99fb-eed48e128397") : secret "insights-runtime-extractor-tls" not found Nov 03 22:12:50.530525 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.530487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:50.530702 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.530567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:50.530702 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.530605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:50.530702 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.530649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:50.530702 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.530657 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 22:12:50.530929 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.530730 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls podName:fdd2ca77-cb6a-4b10-906c-841c51aea0c6 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.530715085 +0000 UTC m=+45.119770753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-hdwl2" (UID: "fdd2ca77-cb6a-4b10-906c-841c51aea0c6") : secret "samples-operator-tls" not found Nov 03 22:12:50.530929 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.530813 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:50.530929 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.530867 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls podName:cf0d1c58-9465-423d-b479-50956f62c7f2 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.530851069 +0000 UTC m=+45.119906738 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v9lq6" (UID: "cf0d1c58-9465-423d-b479-50956f62c7f2") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:50.530929 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.530926 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:12:50.531167 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.530964 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.530951707 +0000 UTC m=+45.120007378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : secret "router-metrics-certs-default" not found Nov 03 22:12:50.531167 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.531029 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.531016094 +0000 UTC m=+45.120071762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : configmap references non-existent config key: service-ca.crt Nov 03 22:12:50.632063 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.632037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:50.632192 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.632110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:50.632192 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.632170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:50.632322 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632231 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:50.632322 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632251 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd57577d7-9mscc: secret "image-registry-tls" not found Nov 03 22:12:50.632322 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632266 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:12:50.632462 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:50.632392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:50.632462 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632427 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:50.632565 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632465 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls podName:d0dc65c0-bcea-4495-bf55-5095632b71b1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.632444664 +0000 UTC m=+45.221500336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls") pod "image-registry-5fd57577d7-9mscc" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1") : secret "image-registry-tls" not found Nov 03 22:12:50.632565 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632492 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert podName:c9994b27-e2c7-4e4a-ab0c-796c19becb5c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.632481697 +0000 UTC m=+45.221537365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zrq7h" (UID: "c9994b27-e2c7-4e4a-ab0c-796c19becb5c") : secret "networking-console-plugin-cert" not found Nov 03 22:12:50.632565 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632512 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.632503523 +0000 UTC m=+45.221559200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:12:50.632565 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632530 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:50.632565 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632542 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655957577d-lgbk6: secret "image-registry-tls" not found Nov 03 22:12:50.632834 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:50.632589 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls podName:01e345d4-1cb8-4e4f-822d-d71b582b53b3 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.632572514 +0000 UTC m=+45.221628200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls") pod "image-registry-655957577d-lgbk6" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3") : secret "image-registry-tls" not found Nov 03 22:12:51.312650 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:51.312610 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g7xxx" event={"ID":"019bf19b-ca0e-4a02-af50-d6450151540e","Type":"ContainerStarted","Data":"7bd827e4928be83698afb81724b540772253a1ca9155fad2099e8605a67fe447"} Nov 03 22:12:51.313054 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:51.312761 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:51.358893 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:51.358839 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g7xxx" podStartSLOduration=34.262646508 podStartE2EDuration="37.358825666s" totalImagesPullingTime="3.096179158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:12:51.357878174 +0000 UTC m=+37.946933863" watchObservedRunningTime="2026-11-03 22:12:51.358825666 +0000 UTC m=+37.947881365" Nov 03 22:12:52.450449 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:52.450413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:52.450920 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:52.450471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:52.450920 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:52.450560 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 22:12:52.450920 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:52.450654 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:12:52.450920 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:52.450681 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm podName:5223b025-80ca-4412-99fb-eed48e128397 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.450664122 +0000 UTC m=+43.039719790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7vs67" (UID: "5223b025-80ca-4412-99fb-eed48e128397") : configmap "kube-rbac-proxy" not found Nov 03 22:12:52.450920 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:52.450718 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls podName:5223b025-80ca-4412-99fb-eed48e128397 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.450700981 +0000 UTC m=+43.039756650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vs67" (UID: "5223b025-80ca-4412-99fb-eed48e128397") : secret "insights-runtime-extractor-tls" not found Nov 03 22:12:53.139034 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:53.138995 2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:53.139449 ip-10-0-133-193 kubenswrapper[2580]: I1103 22:12:53.139431 2580 scope.go:122] "RemoveContainer" containerID="0a268de110e22153c69a83cfaf9d6fed6ce349dd349ecd27279f24377dc95850" Nov 03 22:12:53.139667 ip-10-0-133-193 kubenswrapper[2580]: E1103 22:12:53.139642 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-xsr82_openshift-console-operator(0c3d06f5-c335-4cf2-a901-cae1f6c2e142)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" podUID="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" Nov 03 22:12:53.539685 ip-10-0-133-193 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 22:12:53.547897 ip-10-0-133-193 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 22:12:53.548063 ip-10-0-133-193 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 22:12:53.548109 ip-10-0-133-193 systemd[1]: kubelet.service: Consumed 2.951s CPU time. Nov 03 22:12:53.555994 ip-10-0-133-193 systemd[1]: Starting Kubernetes Kubelet... Nov 03 22:12:53.652057 ip-10-0-133-193 kubenswrapper[4903]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:53.652057 ip-10-0-133-193 kubenswrapper[4903]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 22:12:53.652057 ip-10-0-133-193 kubenswrapper[4903]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:53.652057 ip-10-0-133-193 kubenswrapper[4903]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 22:12:53.654192 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654110 4903 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:53.654192 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654192 4903 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:53.654192 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654196 4903 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654199 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654202 4903 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654204 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654207 4903 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654211 4903 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654214 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654216 4903 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654219 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654222 4903 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654224 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654227 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654231 4903 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654236 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654239 4903 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654242 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654252 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654255 4903 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654258 4903 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:53.654310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654260 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654263 4903 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654266 4903 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654269 4903 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654272 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654274 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654277 4903 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654280 4903 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654282 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654285 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654287 4903 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654309 4903 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654312 4903 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654315 4903 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654318 4903 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654321 4903 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654324 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654327 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654329 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654332 4903 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:53.654772 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654334 4903 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654337 4903 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654340 4903 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654342 4903 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654345 4903 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654348 4903 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654351 4903 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654354 4903 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654357 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654359 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654362 4903 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654365 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654367 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654371 4903 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654374 4903 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654376 4903 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654379 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654382 4903 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654385 4903 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:53.655308 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654387 4903 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654390 4903 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654393 4903 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654397 4903 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654402 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654405 4903 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654410 4903 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654413 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654417 4903 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654421 4903 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654425 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654429 4903 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654434 4903 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654438 4903 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654442 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654444 4903 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654447 4903 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654450 4903 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654453 4903 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654455 4903 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:53.655777 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654458 4903 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654461 4903 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654464 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654468 4903 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654471 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654474 4903 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654476 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654479 4903 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654482 4903 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654484 4903 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654487 4903 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654489 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654492 4903 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654495 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654497 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654500 4903 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654502 4903 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654505 4903 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654509 4903 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:53.656262 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654515 4903 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654519 4903 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654522 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654526 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654528 4903 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654532 4903 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.654534 4903 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655312 4903 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655319 4903 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655322 4903 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655326 4903 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655328 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655331 4903 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655334 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655337 4903 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655341 4903 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655344 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655346 4903 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655355 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:53.656733 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655359 4903 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655361 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655364 4903 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655367 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655370 4903 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655373 4903 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655376 4903 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655379 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655381 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655384 4903 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655386 4903 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655389 4903 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655392 4903 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655395 4903 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655397 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655400 4903 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655402 4903 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655405 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655407 4903 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655410 4903 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:53.657190 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655413 4903 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655416 4903 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655418 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655421 4903 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655426 4903 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655430 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655434 4903 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655436 4903 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655439 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655444 4903 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655446 4903 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655449 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655452 4903 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655454 4903 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655457 4903 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655460 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655463 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655465 4903 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655468 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:53.657695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655470 4903 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655473 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655476 4903 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655478 4903 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655481 4903 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655483 4903 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655486 4903 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655488 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655491 4903 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655493 4903 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655496 4903 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655498 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655505 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655507 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655511 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655513 4903 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655516 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655518 4903 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655521 4903 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655524 4903 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:53.658157 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655526 4903 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655530 4903 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655534 4903 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655538 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655542 4903 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655544 4903 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655547 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655550 4903 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655552 4903 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655555 4903 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655557 4903 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655560 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655562 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655565 4903 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655568 4903 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655570 4903 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655573 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655575 4903 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655578 4903 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655581 4903 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:53.658656 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655583 4903 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655586 4903 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655589 4903 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655591 4903 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655595 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655598 4903 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655600 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655603 4903 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655606 4903 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655608 4903 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655611 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655613 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655616 4903 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655619 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.655621 4903 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656114 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656138 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656143 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656146 4903 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:53.659140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656150 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656152 4903 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656156 4903 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656160 4903 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656164 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656167 4903 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656169 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656172 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656174 4903 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656177 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656180 4903 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656182 4903 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656185 4903 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656187 4903 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656190 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656193 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656195 4903 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656200 4903 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656203 4903 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:53.659634 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656206 4903 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656208 4903 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656211 4903 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656214 4903 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656217 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656219 4903 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656222 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656225 4903 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656227 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656230 4903 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656233 4903 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656236 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656238 4903 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656241 4903 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656244 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656247 4903 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656249 4903 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656252 4903 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656254 4903 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656257 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:53.660283 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656259 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656262 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656264 4903 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656267 4903 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656270 4903 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656272 4903 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656275 4903 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656278 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656280 4903 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656284 4903 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656288 4903 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656303 4903 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656306 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656309 4903 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656311 4903 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656314 4903 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656316 4903 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656319 4903 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656322 4903 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:53.661205 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656324 4903 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656327 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656330 4903 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656332 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656335 4903 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656338 4903 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656340 4903 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656343 4903 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656347 4903 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656351 4903 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656355 4903 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656358 4903 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656362 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656365 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656368 4903 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656371 4903 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656374 4903 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656377 4903 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656379 4903 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656382 4903 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:53.662031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656384 4903 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656387 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656390 4903 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656394 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656396 4903 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656399 4903 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656402 4903 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656404 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656407 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656410 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656413 4903 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656415 4903 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656418 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656421 4903 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656423 4903 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656426 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656429 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656431 4903 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656434 4903 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:53.662935 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656436 4903 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656440 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656443 4903 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656445 4903 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.656448 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656542 4903 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656559 4903 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656568 4903 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656574 4903 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656581 4903 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656586 4903 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656591 4903 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656600 4903 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656603 4903 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656606 4903 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656610 4903 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656614 4903 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656618 4903 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656621 4903 flags.go:64] FLAG: --cgroup-root="" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656624 4903 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656627 4903 flags.go:64] FLAG: --client-ca-file="" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656630 4903 flags.go:64] FLAG: --cloud-provider="external" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656633 4903 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 22:12:53.663862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656637 4903 flags.go:64] FLAG: --cluster-domain="" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656640 4903 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656643 4903 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656646 4903 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656649 4903 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656653 4903 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656656 4903 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656659 4903 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656663 4903 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656666 4903 flags.go:64] FLAG: --contention-profiling="false" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656674 4903 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656677 4903 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656681 4903 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656684 4903 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656688 4903 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656691 4903 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656694 4903 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656697 4903 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656700 4903 flags.go:64] FLAG: --enable-server="true" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656703 4903 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656708 4903 flags.go:64] FLAG: --event-burst="100" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656711 4903 flags.go:64] FLAG: --event-qps="50" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656714 4903 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656717 4903 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 22:12:53.664862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656720 4903 flags.go:64] FLAG: --eviction-hard="" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656726 4903 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656729 4903 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656732 4903 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656736 4903 flags.go:64] FLAG: --eviction-soft="" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656739 4903 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656742 4903 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656745 4903 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656748 4903 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656751 4903 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656754 4903 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656757 4903 flags.go:64] FLAG: --feature-gates="" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656761 4903 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656764 4903 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656768 4903 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656771 4903 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656774 4903 flags.go:64] FLAG: --healthz-port="10248" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656778 4903 flags.go:64] FLAG: --help="false" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656783 4903 flags.go:64] FLAG: --hostname-override="ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656786 4903 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656789 4903 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656792 4903 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656799 4903 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 22:12:53.665923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656802 4903 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656805 4903 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656808 4903 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656811 4903 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656814 4903 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656818 4903 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656821 4903 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656824 4903 flags.go:64] FLAG: --kube-reserved="" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656827 4903 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656830 4903 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656834 4903 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656837 4903 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656840 4903 flags.go:64] FLAG: --lock-file="" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656842 4903 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656845 4903 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656848 4903 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656854 4903 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656857 4903 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656860 4903 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656862 4903 flags.go:64] FLAG: --logging-format="text" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656865 4903 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656869 4903 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656872 4903 flags.go:64] FLAG: --manifest-url="" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656874 4903 flags.go:64] FLAG: --manifest-url-header="" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656879 4903 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 22:12:53.666925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656882 4903 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656886 4903 flags.go:64] FLAG: --max-pods="110" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656891 4903 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656895 4903 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656898 4903 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656900 4903 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656904 4903 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656906 4903 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656909 4903 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656916 4903 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656920 4903 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656923 4903 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656926 4903 flags.go:64] FLAG: --pod-cidr="" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656928 4903 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656931 4903 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656935 4903 flags.go:64] FLAG: --pods-per-core="0" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656938 4903 flags.go:64] FLAG: --port="10250" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656942 4903 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656945 4903 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02ae7526c9f2e8caa" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656948 4903 flags.go:64] FLAG: --qos-reserved="" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656951 4903 flags.go:64] FLAG: --read-only-port="10255" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656954 4903 flags.go:64] FLAG: --register-node="true" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656957 4903 flags.go:64] FLAG: --register-with-taints="" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656961 4903 flags.go:64] FLAG: --registry-burst="10" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656964 4903 flags.go:64] FLAG: --registry-qps="5" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656967 4903 flags.go:64] FLAG: --reserved-cpus="" Nov 03 22:12:53.667982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656970 4903 flags.go:64] FLAG: --reserved-memory="" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656974 4903 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656977 4903 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656980 4903 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656983 4903 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656986 4903 flags.go:64] FLAG: --runonce="false" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656989 4903 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656992 4903 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.656997 4903 flags.go:64] FLAG: --seccomp-default="false" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657001 4903 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657004 4903 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657007 4903 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657010 4903 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657013 4903 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657016 4903 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657019 4903 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657023 4903 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657026 4903 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657029 4903 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657032 4903 flags.go:64] FLAG: --system-cgroups="" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657035 4903 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657040 4903 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657043 4903 flags.go:64] FLAG: --tls-cert-file="" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657046 4903 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657050 4903 flags.go:64] FLAG: --tls-min-version="" Nov 03 22:12:53.669088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657053 4903 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657055 4903 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657058 4903 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657061 4903 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657064 4903 flags.go:64] FLAG: --v="2" Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657068 4903 flags.go:64] FLAG: --version="false" Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657072 4903 flags.go:64] FLAG: --vmodule="" Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657076 4903 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657080 4903 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657232 4903 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657236 4903 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657240 4903 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657243 4903 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657246 4903 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657248 4903 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657252 4903 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657255 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657258 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657261 4903 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657264 4903 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657267 4903 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:53.669823 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657269 4903 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657272 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657275 4903 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657277 4903 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657280 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657283 4903 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657285 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657288 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657303 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657306 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657309 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657312 4903 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657315 4903 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657317 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657320 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657323 4903 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657325 4903 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657328 4903 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657330 4903 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657333 4903 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:53.670363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657335 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657338 4903 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657341 4903 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657343 4903 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657346 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657348 4903 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657353 4903 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657355 4903 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657358 4903 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657361 4903 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657363 4903 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657366 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657369 4903 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657371 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657375 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657378 4903 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657381 4903 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657383 4903 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657386 4903 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657389 4903 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:53.671038 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657392 4903 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657394 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657397 4903 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657399 4903 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657402 4903 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657405 4903 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657407 4903 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657410 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657413 4903 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657415 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657418 4903 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657420 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657423 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657425 4903 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657428 4903 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657431 4903 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657435 4903 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657439 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657443 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657446 4903 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:53.671826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657449 4903 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657451 4903 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657454 4903 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657457 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657460 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657463 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657467 4903 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657470 4903 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657474 4903 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657478 4903 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657481 4903 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657484 4903 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657487 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657490 4903 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657493 4903 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657496 4903 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657498 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657501 4903 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657503 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:53.672382 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657506 4903 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657509 4903 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657511 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657514 4903 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657516 4903 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657519 4903 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657522 4903 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657524 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657527 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657529 4903 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657532 4903 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657536 4903 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657539 4903 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657541 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657544 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.657571 4903 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:53.672862 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657606 4903 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657611 4903 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657614 4903 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657617 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657621 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657624 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657626 4903 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657630 4903 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657634 4903 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657637 4903 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657640 4903 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657643 4903 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657646 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657648 4903 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657651 4903 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657653 4903 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657656 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657659 4903 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657661 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:53.673306 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657664 4903 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657667 4903 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657669 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657672 4903 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657674 4903 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657677 4903 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657680 4903 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657682 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657686 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657689 4903 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657691 4903 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657694 4903 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657697 4903 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657699 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657703 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657706 4903 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657708 4903 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657712 4903 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657715 4903 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:53.673894 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657718 4903 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657721 4903 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657723 4903 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657726 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657728 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657731 4903 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657734 4903 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657736 4903 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657739 4903 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657742 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657744 4903 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657747 4903 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657750 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657752 4903 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657755 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657757 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657760 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657762 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657765 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657768 4903 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:53.674403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657771 4903 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657775 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657777 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657780 4903 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657783 4903 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657785 4903 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657788 4903 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657791 4903 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657794 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657796 4903 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657800 4903 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657803 4903 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657806 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657808 4903 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657811 4903 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657813 4903 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657816 4903 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657819 4903 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657821 4903 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657824 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:53.674885 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657827 4903 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657830 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657832 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657835 4903 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657838 4903 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657840 4903 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657843 4903 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657845 4903 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657848 4903 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657851 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657853 4903 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657856 4903 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657858 4903 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657862 4903 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657865 4903 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657867 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657870 4903 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657872 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657875 4903 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657877 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:53.675553 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657880 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657883 4903 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657887 4903 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657890 4903 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657894 4903 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657899 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657902 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.657905 4903 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.662476 4903 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.662496 4903 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.662520 4903 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.662528 4903 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662611 4903 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662621 4903 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662627 4903 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662634 4903 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662639 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662644 4903 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662651 4903 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:53.676026 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662657 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662663 4903 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662669 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662675 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662680 4903 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662684 4903 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662688 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662693 4903 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662698 4903 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662702 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662707 4903 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662711 4903 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662715 4903 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662720 4903 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662725 4903 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662729 4903 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662734 4903 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662738 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662743 4903 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662748 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:53.676504 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662752 4903 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662757 4903 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662762 4903 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662767 4903 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662772 4903 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662776 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662780 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662785 4903 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662789 4903 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662793 4903 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662799 4903 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662803 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662807 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662812 4903 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662816 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662821 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662826 4903 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662830 4903 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662834 4903 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662839 4903 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:53.676984 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662844 4903 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662848 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662852 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662856 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662860 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662866 4903 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662871 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662875 4903 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662880 4903 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662884 4903 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662889 4903 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662893 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662898 4903 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662902 4903 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662906 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662912 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662916 4903 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662920 4903 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662925 4903 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:53.677523 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662929 4903 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662933 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662937 4903 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662941 4903 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662945 4903 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662950 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662954 4903 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662959 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662963 4903 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662967 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662972 4903 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662976 4903 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662981 4903 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662986 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662990 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662995 4903 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.662999 4903 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663004 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663008 4903 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663013 4903 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:53.677978 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663018 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663022 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663026 4903 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663031 4903 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663036 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663040 4903 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663044 4903 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663048 4903 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663053 4903 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663058 4903 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663063 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663067 4903 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663071 4903 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663075 4903 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663080 4903 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663084 4903 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663088 4903 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663092 4903 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663097 4903 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:53.678473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663102 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.663143 4903 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663384 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663395 4903 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663403 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663408 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663412 4903 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663418 4903 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663424 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663430 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663435 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663439 4903 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663444 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663448 4903 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663453 4903 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:53.679011 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663458 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663462 4903 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663466 4903 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663471 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663475 4903 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663480 4903 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663484 4903 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663488 4903 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663499 4903 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663504 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663508 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663512 4903 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663517 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663521 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663525 4903 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663530 4903 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663534 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663539 4903 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663543 4903 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663547 4903 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:53.679567 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663551 4903 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663555 4903 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663560 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663564 4903 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663568 4903 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663573 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663577 4903 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663581 4903 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663586 4903 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663590 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663594 4903 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663599 4903 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663604 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663608 4903 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663613 4903 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663617 4903 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663621 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663625 4903 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663630 4903 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663634 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:53.680048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663638 4903 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663644 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663649 4903 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663653 4903 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663657 4903 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663661 4903 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663666 4903 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663670 4903 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663674 4903 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663678 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663683 4903 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663689 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663694 4903 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663699 4903 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663703 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663708 4903 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663712 4903 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663716 4903 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663721 4903 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663725 4903 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:53.680535 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663729 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663733 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663738 4903 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663742 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663746 4903 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663751 4903 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663756 4903 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663760 4903 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663765 4903 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663769 4903 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663773 4903 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663777 4903 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663781 4903 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663785 4903 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663791 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663795 4903 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663799 4903 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663804 4903 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663808 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663812 4903 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:53.681015 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663817 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663821 4903 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663826 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663830 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663837 4903 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663842 4903 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663846 4903 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663850 4903 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663855 4903 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663859 4903 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663863 4903 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663867 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663871 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.663913 4903 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663965 4903 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 22:12:53.681637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663972 4903 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663976 4903 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663981 4903 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663986 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663990 4903 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663994 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.663999 4903 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664003 4903 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664007 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664011 4903 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664016 4903 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664020 4903 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664025 4903 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664030 4903 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664034 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664039 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664043 4903 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664047 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664051 4903 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664055 4903 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 22:12:53.682104 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664060 4903 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664065 4903 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664071 4903 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664075 4903 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664080 4903 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664084 4903 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664089 4903 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664093 4903 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664097 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664102 4903 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664106 4903 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664111 4903 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664115 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664119 4903 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664124 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664129 4903 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664133 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664137 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664141 4903 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 22:12:53.682760 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664147 4903 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664151 4903 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664156 4903 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664160 4903 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664164 4903 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664168 4903 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664173 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664177 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664181 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664187 4903 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664194 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664199 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664203 4903 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664210 4903 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664215 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664222 4903 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664227 4903 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664231 4903 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 22:12:53.683414 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664236 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664240 4903 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664244 4903 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664249 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664253 4903 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664258 4903 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664262 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664267 4903 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664271 4903 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664276 4903 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664280 4903 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664284 4903 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664289 4903 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664310 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664315 4903 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664319 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664323 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664327 4903 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664331 4903 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664336 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 22:12:53.683866 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664340 4903 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664345 4903 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664350 4903 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664354 4903 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664359 4903 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664363 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664369 4903 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664373 4903 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664377 4903 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664383 4903 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664387 4903 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664392 4903 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664396 4903 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664401 4903 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664406 4903 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664411 4903 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664415 4903 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664419 4903 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664423 4903 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664428 4903 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 22:12:53.684386 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664432 4903 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664436 4903 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664441 4903 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664445 4903 feature_gate.go:386] unrecognized feature gate: Example Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664449 4903 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664456 4903 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664462 4903 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:53.664466 4903 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.664539 4903 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.666025 4903 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.666092 4903 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.666369 4903 server.go:1014] "Starting client certificate rotation" Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.666499 4903 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.666542 4903 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 22:07:13 +0000 UTC" deadline="2028-05-27 17:17:12.469766278 +0000 UTC" Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.666568 4903 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13699h4m18.8032005s" Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.667080 4903 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:12:53.684933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.668835 4903 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 22:12:53.685465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.671444 4903 log.go:25] "Validated CRI v1 runtime API" Nov 03 22:12:53.685465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.673065 4903 log.go:25] "Validated CRI v1 image API" Nov 03 22:12:53.685465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.673785 4903 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 22:12:53.685465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.675591 4903 fs.go:135] Filesystem UUIDs: map[2decf6e9-2b51-441f-9eed-d592e317b8de:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 85c124c2-90a8-432a-bdbd-e08b0c025042:/dev/nvme0n1p4] Nov 03 22:12:53.685601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.675611 4903 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0df8cc8c1090d72466ae22675707c566e54249f1899243b79363e6eca1a92334/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0df8cc8c1090d72466ae22675707c566e54249f1899243b79363e6eca1a92334/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25a25c5c4a88c1d74521e00b53636f628730d545ff52cb48efa3c693550472c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25a25c5c4a88c1d74521e00b53636f628730d545ff52cb48efa3c693550472c6/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/459aa21bbbcdd85abcc8468397f971c9543d8e8f18ad39baafd7a2a5d87b2a44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/459aa21bbbcdd85abcc8468397f971c9543d8e8f18ad39baafd7a2a5d87b2a44/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bfc8057990bcef4eca76b5c95505191cbbd57978cd478ab8bd9b1006d77926f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bfc8057990bcef4eca76b5c95505191cbbd57978cd478ab8bd9b1006d77926f/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5817d8adba03a158b2d68586b18d2985fa0a10ac1ddcf217e92ebf9f2d5fb732/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5817d8adba03a158b2d68586b18d2985fa0a10ac1ddcf217e92ebf9f2d5fb732/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f2f55b8f9921029a65e5aa9814552ecefb8a64bf4564f6ce296a77d21718518/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f2f55b8f9921029a65e5aa9814552ecefb8a64bf4564f6ce296a77d21718518/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60a0060ae6f7a4d82e977cd8dea8ea73a2c80457a95a24ddc4acf88a682a2172/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60a0060ae6f7a4d82e977cd8dea8ea73a2c80457a95a24ddc4acf88a682a2172/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65ebbaf992317bd2929513c86b7b6287bb273fd789194acec14e6381404a127c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65ebbaf992317bd2929513c86b7b6287bb273fd789194acec14e6381404a127c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fec7e3cb2ca04b9eb72c015cddf8c6d390ba71d13dd7e74b83861dad88e8482/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fec7e3cb2ca04b9eb72c015cddf8c6d390ba71d13dd7e74b83861dad88e8482/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c6a86ac3f8b52977e833b5ad355764cfd791740e99e210c54aff67296ebb2ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c6a86ac3f8b52977e833b5ad355764cfd791740e99e210c54aff67296ebb2ca/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bc09fba35c7598c9b4fe4cfe87d932b6b13d539014543558f0e99256cd5a770/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bc09fba35c7598c9b4fe4cfe87d932b6b13d539014543558f0e99256cd5a770/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/922f69908bf42654c7cff9ac12ab13fc008bd97a8480ad6253c5e970184b5263/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/922f69908bf42654c7cff9ac12ab13fc008bd97a8480ad6253c5e970184b5263/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93e9e80e1ff802ecac50970c92c0dcc41bfc164f371eefafc5fce32c07a913ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93e9e80e1ff802ecac50970c92c0dcc41bfc164f371eefafc5fce32c07a913ad/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4aad4e5a91a2b20a8df1381f665061481cbf65bd6a11b7cb909599fed022247/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4aad4e5a91a2b20a8df1381f665061481cbf65bd6a11b7cb909599fed022247/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b53ac119811039de567e6ea6c10de3dea98d2d45595c38fc0371e4085ca7af42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b53ac119811039de567e6ea6c10de3dea98d2d45595c38fc0371e4085ca7af42/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8820d4b769ead6d669adc81d3580ddb6591a9077473b7dc56ee136349bd8461/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8820d4b769ead6d669adc81d3580ddb6591a9077473b7dc56ee136349bd8461/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9ddb67e3dd68da22f612d7158c8e0b43ad1b0970792439fa6e694b53a2256b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9ddb67e3dd68da22f612d7158c8e0b43ad1b0970792439fa6e694b53a2256b3/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d50feafba7e23b2aa8fb541005ebc7acaa6448643b3bf15b9843323258174be9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d50feafba7e23b2aa8fb541005ebc7acaa6448643b3bf15b9843323258174be9/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d83bfeb13bebe06dc72d8cd05482d56d69b5c83ab4f92008ac8962ba1f1bbb70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d83bfeb13bebe06dc72d8cd05482d56d69b5c83ab4f92008ac8962ba1f1bbb70/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db7c2c0c5dee19fe354cad7fc4a0db4719d8ac45ad8aa898643221673a90abf6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db7c2c0c5dee19fe354cad7fc4a0db4719d8ac45ad8aa898643221673a90abf6/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec3ddd1cfa9e74d5b968c59a676d4beae48b19ece3bb527da0d1d619dbf8d738/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec3ddd1cfa9e74d5b968c59a676d4beae48b19ece3bb527da0d1d619dbf8d738/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8cd69b5fc7b0847591e23f59b2422bd6a7cf573e8f845423eb620d34750e09b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8cd69b5fc7b0847591e23f59b2422bd6a7cf573e8f845423eb620d34750e09b/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/019bf19b-ca0e-4a02-af50-d6450151540e/volumes/kubernetes.io~projected/kube-api-access-6sp7f:{mountpoint:/var/lib/kubelet/pods/019bf19b-ca0e-4a02-af50-d6450151540e/volumes/kubernetes.io~projected/kube-api-access-6sp7f major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~projected/kube-api-access-jq4nt:{mountpoint:/var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~projected/kube-api-access-jq4nt major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a18566a-0818-494c-989b-da2abea7edd9/volumes/kubernetes.io~projected/kube-api-access-pvxxf:{mountpoint:/var/lib/kubelet/pods/0a18566a-0818-494c-989b-da2abea7edd9/volumes/kubernetes.io~projected/kube-api-access-pvxxf major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a18566a-0818-494c-989b-da2abea7edd9/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/0a18566a-0818-494c-989b-da2abea7edd9/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a18566a-0818-494c-989b-da2abea7edd9/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/0a18566a-0818-494c-989b-da2abea7edd9/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bdc4254-64f5-4613-b007-85bc708b2d9b/volumes/kubernetes.io~projected/kube-api-access-2nwmt:{mountpoint:/var/lib/kubelet/pods/0bdc4254-64f5-4613-b007-85bc708b2d9b/volumes/kubernetes.io~projected/kube-api-access-2nwmt major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bdc4254-64f5-4613-b007-85bc708b2d9b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0bdc4254-64f5-4613-b007-85bc708b2d9b/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c3d06f5-c335-4cf2-a901-cae1f6c2e142/volumes/kubernetes.io~projected/kube-api-access-8xlx2:{mountpoint:/var/lib/kubelet/pods/0c3d06f5-c335-4cf2-a901-cae1f6c2e142/volumes/kubernetes.io~projected/kube-api-access-8xlx2 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c3d06f5-c335-4cf2-a901-cae1f6c2e142/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0c3d06f5-c335-4cf2-a901-cae1f6c2e142/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ac77e3c-256e-4a95-8106-de9bda8864c5/volumes/kubernetes.io~projected/kube-api-access-j8hmn:{mountpoint:/var/lib/kubelet/pods/2ac77e3c-256e-4a95-8106-de9bda8864c5/volumes/kubernetes.io~projected/kube-api-access-j8hmn major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/305d6b5c-15d1-4c7e-8b31-db2c71402cd3/volumes/kubernetes.io~projected/kube-api-access-fzpl5:{mountpoint:/var/lib/kubelet/pods/305d6b5c-15d1-4c7e-8b31-db2c71402cd3/volumes/kubernetes.io~projected/kube-api-access-fzpl5 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33d39b02-7a8c-4edc-ae53-3ebd1e494f58/volumes/kubernetes.io~projected/kube-api-access-ck4pl:{mountpoint:/var/lib/kubelet/pods/33d39b02-7a8c-4edc-ae53-3ebd1e494f58/volumes/kubernetes.io~projected/kube-api-access-ck4pl major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33d39b02-7a8c-4edc-ae53-3ebd1e494f58/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/33d39b02-7a8c-4edc-ae53-3ebd1e494f58/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/375eaed5-066a-4615-bc60-09e343aec21b/volumes/kubernetes.io~projected/kube-api-access-b25gv:{mountpoint:/var/lib/kubelet/pods/375eaed5-066a-4615-bc60-09e343aec21b/volumes/kubernetes.io~projected/kube-api-access-b25gv major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/375eaed5-066a-4615-bc60-09e343aec21b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/375eaed5-066a-4615-bc60-09e343aec21b/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44c8b7d7-71ad-47ee-9aad-be7f366974f5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/44c8b7d7-71ad-47ee-9aad-be7f366974f5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44c8b7d7-71ad-47ee-9aad-be7f366974f5/volumes/kubernetes.io~projected/kube-api-access-987t6:{mountpoint:/var/lib/kubelet/pods/44c8b7d7-71ad-47ee-9aad-be7f366974f5/volumes/kubernetes.io~projected/kube-api-access-987t6 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44c8b7d7-71ad-47ee-9aad-be7f366974f5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/44c8b7d7-71ad-47ee-9aad-be7f366974f5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5223b025-80ca-4412-99fb-eed48e128397/volumes/kubernetes.io~projected/kube-api-access-shqjf:{mountpoint:/var/lib/kubelet/pods/5223b025-80ca-4412-99fb-eed48e128397/volumes/kubernetes.io~projected/kube-api-access-shqjf major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60151974-97b1-4371-9831-762b096bc3fb/volumes/kubernetes.io~projected/kube-api-access-bz9qh:{mountpoint:/var/lib/kubelet/pods/60151974-97b1-4371-9831-762b096bc3fb/volumes/kubernetes.io~projected/kube-api-access-bz9qh major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60151974-97b1-4371-9831-762b096bc3fb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/60151974-97b1-4371-9831-762b096bc3fb/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63867b60-8f1a-44a6-bbf1-8e521d232081/volumes/kubernetes.io~projected/kube-api-access-vmw5v:{mountpoint:/var/lib/kubelet/pods/63867b60-8f1a-44a6-bbf1-8e521d232081/volumes/kubernetes.io~projected/kube-api-access-vmw5v major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/677c391f-bc16-46c8-9439-02a1f3bf934d/volumes/kubernetes.io~projected/kube-api-access-dmrwn:{mountpoint:/var/lib/kubelet/pods/677c391f-bc16-46c8-9439-02a1f3bf934d/volumes/kubernetes.io~projected/kube-api-access-dmrwn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c35b7b2-f209-4ff3-ab43-d8533fcdf341/volumes/kubernetes.io~projected/kube-api-access-6cqs8:{mountpoint:/var/lib/kubelet/pods/7c35b7b2-f209-4ff3-ab43-d8533fcdf341/volumes/kubernetes.io~projected/kube-api-access-6cqs8 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/805b7a3b-03f1-4799-a283-235314973feb/volumes/kubernetes.io~projected/kube-api-access-2c574:{mountpoint:/var/lib/kubelet/pods/805b7a3b-03f1-4799-a283-235314973feb/volumes/kubernetes.io~projected/kube-api-access-2c574 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fc6de3b-56b8-4344-b7ab-3ad21f645803/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8fc6de3b-56b8-4344-b7ab-3ad21f645803/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fc6de3b-56b8-4344-b7ab-3ad21f645803/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8fc6de3b-56b8-4344-b7ab-3ad21f645803/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fc6de3b-56b8-4344-b7ab-3ad21f645803/volumes/kubernetes.io~projected/kube-api-access-v9lch:{mountpoint:/var/lib/kubelet/pods/8fc6de3b-56b8-4344-b7ab-3ad21f645803/volumes/kubernetes.io~projected/kube-api-access-v9lch major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95/volumes/kubernetes.io~projected/kube-api-access-5ftn6:{mountpoint:/var/lib/kubelet/pods/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95/volumes/kubernetes.io~projected/kube-api-access-5ftn6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2dea624-bc31-43e7-8691-00c686c3eacc/volumes/kubernetes.io~projected/kube-api-access-759px:{mountpoint:/var/lib/kubelet/pods/a2dea624-bc31-43e7-8691-00c686c3eacc/volumes/kubernetes.io~projected/kube-api-access-759px major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a714258f-ba4e-4792-853b-7d20411d3ca7/volumes/kubernetes.io~projected/kube-api-access-62rmb:{mountpoint:/var/lib/kubelet/pods/a714258f-ba4e-4792-853b-7d20411d3ca7/volumes/kubernetes.io~projected/kube-api-access-62rmb major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcefadd9-c607-447d-85ce-fe31bc85e875/volumes/kubernetes.io~projected/kube-api-access-27xh4:{mountpoint:/var/lib/kubelet/pods/bcefadd9-c607-447d-85ce-fe31bc85e875/volumes/kubernetes.io~projected/kube-api-access-27xh4 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fa970-77b7-4f41-9a2d-a586343c51da/volumes/kubernetes.io~projected/kube-api-access-7rhtc:{mountpoint:/var/lib/kubelet/pods/bf8fa970-77b7-4f41-9a2d-a586343c51da/volumes/kubernetes.io~projected/kube-api-access-7rhtc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1b7053e-001b-4d96-9bf3-1a768e364346/volumes/kubernetes.io~projected/kube-api-access-9t9hd:{mountpoint:/var/lib/kubelet/pods/c1b7053e-001b-4d96-9bf3-1a768e364346/volumes/kubernetes.io~projected/kube-api-access-9t9hd major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf0d1c58-9465-423d-b479-50956f62c7f2/volumes/kubernetes.io~projected/kube-api-access-54ggj:{mountpoint:/var/lib/kubelet/pods/cf0d1c58-9465-423d-b479-50956f62c7f2/volumes/kubernetes.io~projected/kube-api-access-54ggj major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~projected/kube-api-access-fskmm:{mountpoint:/var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~projected/kube-api-access-fskmm major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb/volumes/kubernetes.io~projected/kube-api-access-cx28g:{mountpoint:/var/lib/kubelet/pods/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb/volumes/kubernetes.io~projected/kube-api-access-cx28g major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e881588e-ae98-4f04-947d-1a765b8a4e5e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e881588e-ae98-4f04-947d-1a765b8a4e5e/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdd2ca77-cb6a-4b10-906c-841c51aea0c6/volumes/kubernetes.io~projected/kube-api-access-rzpjd:{mountpoint:/var/lib/kubelet/pods/fdd2ca77-cb6a-4b10-906c-841c51aea0c6/volumes/kubernetes.io~projected/kube-api-access-rzpjd major:0 minor:204 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c89ed5b7b366b0e71526b6fd715fdc92005f10e96c3539bdb371689054c5a310/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/190afc1aab8e440c956c0c0672286f0ef174fb5af208461e481457660d732b8d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7a038a5fc8950b7b3dbe5b3319d191a1b51598ab4e7c75e6ef303230a8a93faa/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/51a0f3e9fe7cc4ca40f0a98a503ada0a08c927fb6c32ca9d159e27a8f92489b3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/66c08ce1d3608694351a070bc245e6ad7b8e8f346e40b80e3303c9829d6be34b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ec3ec15f6838be42970f247223d8691b92420b72c7c91c7ffd8cbc6fe0210624/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/40f698fd4fefd4c02b1288bf73e3a153d5a5abccc417ac0abfb47443c0fdb2c3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f611fd1a91b5249419a0f6748fe8ad32e058061de803b12e93de60757d57f171/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6492f2bceab22d8672a7902c603311ec8660c1940cf81379a9f148b5890fdfe0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2568f1cb30805d9a773340469e8009f49c0411aed0a2f7743f3793452cfe3eb5/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/70823086b4de05de54a455a06013b03966cd557cc9fe4f3bdae2faa41b2c340c/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/ef8da04f47099608518e9202d622f37474d2ec08e36f0a8edee997b610dacea8/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/78760629a541df7bf572e21f74425663c8cadd790e8c90a24e849d452a3f4587/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9d5eb2e24533aa844b6996832e76a78cea9f06af629d88db9b7eb2ba40b5d37d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9b7e001d410bb87401407bad2ef5ce9c27211e4244dd7b23ffa4864473b14c5b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2f25b3a31cf1093056d1a086d8eb1c3d8ceba6c274391002b237283604af317d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/80ccba2b2c704a68acb55d8dacd420394fd62d3fe9ffb28f29a21d489c8f73a8/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/749e8d8fb06e351447d6b7419c168c2ddf2c9f6214f10614780ee4db81779043/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/24dd9f1521c08a61bde6b3c7a3dcfa48643f7ea0872d432134efe0a6f57b2143/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/8f14011a827187b514eda15f3936a31102a64b172d38c0c0d24786afc2c85d5e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b83a4cdfe1c10972980f96eefdbefa82f72df718d04031793b4006dcd22d2562/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/ff28d735bc84c76131b2eafc9cd3041c3e1c88e48694ab732b1f7160ce3098c4/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/a886621bf804dc23bc467158d5c834a8faf1eb5f6aac50f5be8d52ae677194a1/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/045211bdd49429e26712589c6021906b2a3914a8b3a5195c4d306ce09f0119c0/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/64e04db2d311ae3d56df09f71ac9fcd460f0fb158b37e78f86f440b53fef9173/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/14d643b4dca4a31880c0f04ea7ff67e25651dcbe640cb00fdee7ffd7a664b9dd/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/7a3dd5b112cf25bfda11b8f70850ab2b50dede0eb3013d0f649bd3be2257fc87/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/2df6710f68ad3cda23c2df9277e130116025ebb6d2381a95f7797540ac735d37/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/fce514091c9df8c2abd66a05d5c88ffc40bef0a91a974b9a78833290448b6868/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/c82c741252e727365ec2bda83d0e622d4eef6e8431f6bb652ff2eaafe329c195/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/7aa79b8b354069f183cb25b9d8e17e3c4d02b9db3a99416190dc236720d96702/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/d8652d51e7f13424657c05d115227b8fbbd749980e2668b32351ec68a7db2380/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b058bfcc3890bc4c51208cc6e21726b15eaf85767868f8079c1d7788f6d979db/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/0b36ac666e85519b4994f0376f3d9dca27b7d28ebc10d345b9c96b1ce340bbfe/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/84468d9e6007dc43b6316ecdd4b55b8e5a04baba164e16f47127e6b91e61b864/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/5c6b564cea47180848472f089d31d6c0e738909eb24b6d3035749e8f54f4fe38/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/63f279fe5f1dd219990ffc1c3a761f29bae8918d394d2ffa7368565cf5927c78/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/bcb5bbca24562238d8944e1b31bca0d18956facd527c0ebfb0db18de0644fedc/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/464dd7025f21f8007b2a46f5367003d0f21915f51f2044eb6417eeadc5172780/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/27ed515cf28001733b7b17f50ec6d76870ac3f46eff1d380591ee27f1c7e3ad1/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/32fa2ab9774b43605f5f2dfd29dceeade5f3281e49396cd5c2b8b8223212a669/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/e92902ec2e7b62769cf1845f94514280cf4c75171a410faa2977672db3f32c40/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/8ab292c790c4c5d9af0e920982d4dbe5151c6b3e126bdcb7a6205b6488da5cce/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/04de8e5d8e10f6c483edf70e84c04bfc110789042e4dc19c7fbafa394540cb4a/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c3277fcb150c62a48743e1a31ee0ff2b381cb8564c416234df3645c229d369e2/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/420e4560e16c95f0c9f509922aaed4f0c8dfec237c1053ec2e1cfe3d7cece9f4/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b72a3f8cef07b12537688bc701eb0167a19dedcef94082e7e4c8d435fcf4b735/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/0be37b3f59d9d2d0ab81a00c020d0997334e5796a716693072056c373e93185b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/3d84f17c3c0bcef3e4e3f5631bbf9a25b7a79ee309b6ba31aa847eb6414dbe25/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b5bfc913581c468b123d3f4922a343e659a6dacfa40856e4d4e197880d3b2e02/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/8354abe6a6c7f653a9f5094e9915ebe90519e8c6f4e21e86b2d3cf590e46db77/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/16579eacdf4d6c274065bea3e7cc722c2a4aaf8cd71f85066545641b2c8e2521/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e1a5b02ca071e2796b1278bd602a3a2848e99d9dd40ebaa8c55e14389367c31d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bf40acfbaf81af37233012eb884521e8c5e1fd6b84e1654c9fd864e85e251497/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.683835 4903 manager.go:217] Machine: {Timestamp:2026-11-03 22:12:53.683376829 +0000 UTC m=+0.087317795 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100046 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28442d25d02e6aedbb44e5377b2707 SystemUUID:ec28442d-25d0-2e6a-edbb-44e5377b2707 BootID:59ec0a2a-c43f-4ec6-80e4-9275aef873c6 Filesystems:[{Device:/run/containers/storage/overlay-containers/7c6a86ac3f8b52977e833b5ad355764cfd791740e99e210c54aff67296ebb2ca/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bfc8057990bcef4eca76b5c95505191cbbd57978cd478ab8bd9b1006d77926f/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44c8b7d7-71ad-47ee-9aad-be7f366974f5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fc6de3b-56b8-4344-b7ab-3ad21f645803/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0df8cc8c1090d72466ae22675707c566e54249f1899243b79363e6eca1a92334/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec3ddd1cfa9e74d5b968c59a676d4beae48b19ece3bb527da0d1d619dbf8d738/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a18566a-0818-494c-989b-da2abea7edd9/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/305d6b5c-15d1-4c7e-8b31-db2c71402cd3/volumes/kubernetes.io~projected/kube-api-access-fzpl5 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93e9e80e1ff802ecac50970c92c0dcc41bfc164f371eefafc5fce32c07a913ad/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/44c8b7d7-71ad-47ee-9aad-be7f366974f5/volumes/kubernetes.io~projected/kube-api-access-987t6 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5817d8adba03a158b2d68586b18d2985fa0a10ac1ddcf217e92ebf9f2d5fb732/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/375eaed5-066a-4615-bc60-09e343aec21b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60151974-97b1-4371-9831-762b096bc3fb/volumes/kubernetes.io~projected/kube-api-access-bz9qh DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/375eaed5-066a-4615-bc60-09e343aec21b/volumes/kubernetes.io~projected/kube-api-access-b25gv DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c3d06f5-c335-4cf2-a901-cae1f6c2e142/volumes/kubernetes.io~projected/kube-api-access-8xlx2 DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/8fc6de3b-56b8-4344-b7ab-3ad21f645803/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33d39b02-7a8c-4edc-ae53-3ebd1e494f58/volumes/kubernetes.io~projected/kube-api-access-ck4pl DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a18566a-0818-494c-989b-da2abea7edd9/volumes/kubernetes.io~projected/kube-api-access-pvxxf DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb/volumes/kubernetes.io~projected/kube-api-access-cx28g DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes/kubernetes.io~projected/kube-api-access-jq4nt DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/8fc6de3b-56b8-4344-b7ab-3ad21f645803/volumes/kubernetes.io~projected/kube-api-access-v9lch DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bc09fba35c7598c9b4fe4cfe87d932b6b13d539014543558f0e99256cd5a770/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d50feafba7e23b2aa8fb541005ebc7acaa6448643b3bf15b9843323258174be9/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95/volumes/kubernetes.io~projected/kube-api-access-5ftn6 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcefadd9-c607-447d-85ce-fe31bc85e875/volumes/kubernetes.io~projected/kube-api-access-27xh4 DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8cd69b5fc7b0847591e23f59b2422bd6a7cf573e8f845423eb620d34750e09b/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/677c391f-bc16-46c8-9439-02a1f3bf934d/volumes/kubernetes.io~projected/kube-api-access-dmrwn DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33d39b02-7a8c-4edc-ae53-3ebd1e494f58/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/922f69908bf42654c7cff9ac12ab13fc008bd97a8480ad6253c5e970184b5263/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d83bfeb13bebe06dc72d8cd05482d56d69b5c83ab4f92008ac8962ba1f1bbb70/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ac77e3c-256e-4a95-8106-de9bda8864c5/volumes/kubernetes.io~projected/kube-api-access-j8hmn DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/019bf19b-ca0e-4a02-af50-d6450151540e/volumes/kubernetes.io~projected/kube-api-access-6sp7f DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db7c2c0c5dee19fe354cad7fc4a0db4719d8ac45ad8aa898643221673a90abf6/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60151974-97b1-4371-9831-762b096bc3fb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0bdc4254-64f5-4613-b007-85bc708b2d9b/volumes/kubernetes.io~projected/kube-api-access-2nwmt DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44c8b7d7-71ad-47ee-9aad-be7f366974f5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b53ac119811039de567e6ea6c10de3dea98d2d45595c38fc0371e4085ca7af42/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c3d06f5-c335-4cf2-a901-cae1f6c2e142/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2dea624-bc31-43e7-8691-00c686c3eacc/volumes/kubernetes.io~projected/kube-api-access-759px DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25a25c5c4a88c1d74521e00b53636f628730d545ff52cb48efa3c693550472c6/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/459aa21bbbcdd85abcc8468397f971c9543d8e8f18ad39baafd7a2a5d87b2a44/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60a0060ae6f7a4d82e977cd8dea8ea73a2c80457a95a24ddc4acf88a682a2172/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cf0d1c58-9465-423d-b479-50956f62c7f2/volumes/kubernetes.io~projected/kube-api-access-54ggj DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fec7e3cb2ca04b9eb72c015cddf8c6d390ba71d13dd7e74b83861dad88e8482/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8820d4b769ead6d669adc81d3580ddb6591a9077473b7dc56ee136349bd8461/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65ebbaf992317bd2929513c86b7b6287bb273fd789194acec14e6381404a127c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4aad4e5a91a2b20a8df1381f665061481cbf65bd6a11b7cb909599fed022247/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bdc4254-64f5-4613-b007-85bc708b2d9b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c1b7053e-001b-4d96-9bf3-1a768e364346/volumes/kubernetes.io~projected/kube-api-access-9t9hd DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fa970-77b7-4f41-9a2d-a586343c51da/volumes/kubernetes.io~projected/kube-api-access-7rhtc DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/805b7a3b-03f1-4799-a283-235314973feb/volumes/kubernetes.io~projected/kube-api-access-2c574 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a18566a-0818-494c-989b-da2abea7edd9/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a714258f-ba4e-4792-853b-7d20411d3ca7/volumes/kubernetes.io~projected/kube-api-access-62rmb DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5223b025-80ca-4412-99fb-eed48e128397/volumes/kubernetes.io~projected/kube-api-access-shqjf DeviceMajor:0 DeviceMinor:306 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f2f55b8f9921029a65e5aa9814552ecefb8a64bf4564f6ce296a77d21718518/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9ddb67e3dd68da22f612d7158c8e0b43ad1b0970792439fa6e694b53a2256b3/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdd2ca77-cb6a-4b10-906c-841c51aea0c6/volumes/kubernetes.io~projected/kube-api-access-rzpjd DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e881588e-ae98-4f04-947d-1a765b8a4e5e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63867b60-8f1a-44a6-bbf1-8e521d232081/volumes/kubernetes.io~projected/kube-api-access-vmw5v DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~projected/kube-api-access-fskmm DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c35b7b2-f209-4ff3-ab43-d8533fcdf341/volumes/kubernetes.io~projected/kube-api-access-6cqs8 DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:459aa21bbbcdd85 MacAddress:f6:9e:e4:15:a1:26 Speed:10000 Mtu:8901} {Name:4bfc8057990bcef MacAddress:36:cf:18:8b:45:24 Speed:10000 Mtu:8901} {Name:5817d8adba03a15 MacAddress:22:1d:f7:c6:60:88 Speed:10000 Mtu:8901} {Name:6fec7e3cb2ca04b MacAddress:22:33:3f:91:a4:bb Speed:10000 Mtu:8901} {Name:7c6a86ac3f8b529 MacAddress:e6:e2:e1:72:d6:01 Speed:10000 Mtu:8901} {Name:8bc09fba35c7598 MacAddress:5e:bd:a1:19:ee:90 Speed:10000 Mtu:8901} {Name:93e9e80e1ff802e MacAddress:4a:01:e2:ca:a0:1b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c4:3f:f8:83 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:93:6c:2a:1e:94 Speed:0 Mtu:8901} {Name:c8820d4b769ead6 MacAddress:6e:14:63:77:3e:70 Speed:10000 Mtu:8901} {Name:c9ddb67e3dd68da MacAddress:be:70:fc:c1:8f:13 Speed:10000 Mtu:8901} {Name:d83bfeb13bebe06 MacAddress:76:6e:31:a7:e9:13 Speed:10000 Mtu:8901} {Name:db7c2c0c5dee19f MacAddress:2a:1e:4b:d2:08:3e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c4:3f:f8:83 Speed:0 Mtu:9001} {Name:f8cd69b5fc7b084 MacAddress:92:1f:7e:7b:c5:7e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:56:ad:13:d5:74:a3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:92:ea:d1:3b:1d:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684154 4903 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684196 4903 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684467 4903 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684483 4903 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684601 4903 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684608 4903 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684618 4903 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684637 4903 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684656 4903 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684806 4903 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684924 4903 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684958 4903 kubelet.go:493] "Attempting to sync node with API server" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684970 4903 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684980 4903 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684990 4903 kubelet.go:405] "Adding apiserver pod source" Nov 03 22:12:53.685821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.684998 4903 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 22:12:53.686553 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.686000 4903 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:12:53.686553 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.686015 4903 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:12:53.686553 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.686525 4903 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 22:12:53.689454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689061 4903 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 22:12:53.689454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689078 4903 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 22:12:53.689454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689215 4903 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 22:12:53.689595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689537 4903 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 22:12:53.689595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689585 4903 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 22:12:53.689849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689836 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 22:12:53.689878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689856 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 22:12:53.689878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689865 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 22:12:53.689878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689874 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 22:12:53.689951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689881 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 22:12:53.689951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689890 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 22:12:53.689951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689899 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 22:12:53.689951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689910 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 22:12:53.689951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689921 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 22:12:53.689951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689929 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 22:12:53.689951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689942 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 22:12:53.690132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689956 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 22:12:53.690132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.689998 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 22:12:53.690132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.690007 4903 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 22:12:53.693387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.693373 4903 server.go:1263] "Started kubelet" Nov 03 22:12:53.693774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.693445 4903 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 22:12:53.693774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.693452 4903 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 22:12:53.693774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.693523 4903 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 22:12:53.693944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.693786 4903 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 22:12:53.693924 ip-10-0-133-193 systemd[1]: Started Kubernetes Kubelet. Nov 03 22:12:53.694557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.694540 4903 server.go:317] "Adding debug handlers to kubelet server" Nov 03 22:12:53.697118 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.697101 4903 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 22:12:53.697118 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.697113 4903 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 22:12:53.698489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.697177 4903 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 22:07:14 +0000 UTC" deadline="2028-06-15 01:38:12.330915195 +0000 UTC" Nov 03 22:12:53.698489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.697210 4903 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14139h25m18.633708585s" Nov 03 22:12:53.698489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.697229 4903 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 22:12:53.698489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.697242 4903 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 22:12:53.698489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.697246 4903 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 22:12:53.698489 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:53.697252 4903 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-193.ec2.internal\" not found" Nov 03 22:12:53.702409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702365 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="375eaed5-066a-4615-bc60-09e343aec21b" volumeName="kubernetes.io/projected/375eaed5-066a-4615-bc60-09e343aec21b-kube-api-access-b25gv" seLinuxMountContext="" Nov 03 22:12:53.702503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702421 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44c8b7d7-71ad-47ee-9aad-be7f366974f5" volumeName="kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-script-lib" seLinuxMountContext="" Nov 03 22:12:53.702503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702436 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44c8b7d7-71ad-47ee-9aad-be7f366974f5" volumeName="kubernetes.io/secret/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 22:12:53.702503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702476 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60151974-97b1-4371-9831-762b096bc3fb" volumeName="kubernetes.io/secret/60151974-97b1-4371-9831-762b096bc3fb-serving-cert" seLinuxMountContext="" Nov 03 22:12:53.702503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702490 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805b7a3b-03f1-4799-a283-235314973feb" volumeName="kubernetes.io/configmap/805b7a3b-03f1-4799-a283-235314973feb-iptables-alerter-script" seLinuxMountContext="" Nov 03 22:12:53.702614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702511 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf0d1c58-9465-423d-b479-50956f62c7f2" volumeName="kubernetes.io/configmap/cf0d1c58-9465-423d-b479-50956f62c7f2-telemetry-config" seLinuxMountContext="" Nov 03 22:12:53.702614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702529 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="019bf19b-ca0e-4a02-af50-d6450151540e" volumeName="kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f" seLinuxMountContext="" Nov 03 22:12:53.702614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702542 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a18566a-0818-494c-989b-da2abea7edd9" volumeName="kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-stats-auth" seLinuxMountContext="" Nov 03 22:12:53.702614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702563 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" volumeName="kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-config" seLinuxMountContext="" Nov 03 22:12:53.702614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702576 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0dc65c0-bcea-4495-bf55-5095632b71b1" volumeName="kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702621 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01e345d4-1cb8-4e4f-822d-d71b582b53b3" volumeName="kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702634 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5223b025-80ca-4412-99fb-eed48e128397" volumeName="kubernetes.io/empty-dir/5223b025-80ca-4412-99fb-eed48e128397-data-volume" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702647 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2dea624-bc31-43e7-8691-00c686c3eacc" volumeName="kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-daemon-config" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702664 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e302a1bc-43db-4e8b-8ed4-f42194f5ddeb" volumeName="kubernetes.io/projected/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-kube-api-access-cx28g" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702681 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e881588e-ae98-4f04-947d-1a765b8a4e5e" volumeName="kubernetes.io/secret/e881588e-ae98-4f04-947d-1a765b8a4e5e-agent-certs" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702698 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05ab17d1-b5dc-4cb6-bf8e-eb650c724aef" volumeName="kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702712 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac77e3c-256e-4a95-8106-de9bda8864c5" volumeName="kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702732 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf0d1c58-9465-423d-b479-50956f62c7f2" volumeName="kubernetes.io/projected/cf0d1c58-9465-423d-b479-50956f62c7f2-kube-api-access-54ggj" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702753 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e302a1bc-43db-4e8b-8ed4-f42194f5ddeb" volumeName="kubernetes.io/configmap/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-config-volume" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702766 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01e345d4-1cb8-4e4f-822d-d71b582b53b3" volumeName="kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702787 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a18566a-0818-494c-989b-da2abea7edd9" volumeName="kubernetes.io/projected/0a18566a-0818-494c-989b-da2abea7edd9-kube-api-access-pvxxf" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702799 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a18566a-0818-494c-989b-da2abea7edd9" volumeName="kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-default-certificate" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702816 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac77e3c-256e-4a95-8106-de9bda8864c5" volumeName="kubernetes.io/projected/2ac77e3c-256e-4a95-8106-de9bda8864c5-kube-api-access-j8hmn" seLinuxMountContext="" Nov 03 22:12:53.702834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702831 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c35b7b2-f209-4ff3-ab43-d8533fcdf341" volumeName="kubernetes.io/projected/7c35b7b2-f209-4ff3-ab43-d8533fcdf341-kube-api-access-6cqs8" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702843 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95" volumeName="kubernetes.io/projected/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-kube-api-access-5ftn6" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702860 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2dea624-bc31-43e7-8691-00c686c3eacc" volumeName="kubernetes.io/projected/a2dea624-bc31-43e7-8691-00c686c3eacc-kube-api-access-759px" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702872 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e302a1bc-43db-4e8b-8ed4-f42194f5ddeb" volumeName="kubernetes.io/empty-dir/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-tmp-dir" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702890 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bdc4254-64f5-4613-b007-85bc708b2d9b" volumeName="kubernetes.io/projected/0bdc4254-64f5-4613-b007-85bc708b2d9b-kube-api-access-2nwmt" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702927 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d39b02-7a8c-4edc-ae53-3ebd1e494f58" volumeName="kubernetes.io/projected/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-kube-api-access-ck4pl" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702941 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fa970-77b7-4f41-9a2d-a586343c51da" volumeName="kubernetes.io/projected/bf8fa970-77b7-4f41-9a2d-a586343c51da-kube-api-access-7rhtc" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702953 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" volumeName="kubernetes.io/secret/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-serving-cert" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702970 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5223b025-80ca-4412-99fb-eed48e128397" volumeName="kubernetes.io/projected/5223b025-80ca-4412-99fb-eed48e128397-kube-api-access-shqjf" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.702983 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01e345d4-1cb8-4e4f-822d-d71b582b53b3" volumeName="kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703001 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bdc4254-64f5-4613-b007-85bc708b2d9b" volumeName="kubernetes.io/secret/0bdc4254-64f5-4613-b007-85bc708b2d9b-cert" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703022 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" volumeName="kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-trusted-ca" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703035 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0dc65c0-bcea-4495-bf55-5095632b71b1" volumeName="kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703053 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01e345d4-1cb8-4e4f-822d-d71b582b53b3" volumeName="kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703066 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="305d6b5c-15d1-4c7e-8b31-db2c71402cd3" volumeName="kubernetes.io/projected/305d6b5c-15d1-4c7e-8b31-db2c71402cd3-kube-api-access-fzpl5" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703083 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d39b02-7a8c-4edc-ae53-3ebd1e494f58" volumeName="kubernetes.io/configmap/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-config" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703096 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805b7a3b-03f1-4799-a283-235314973feb" volumeName="kubernetes.io/projected/805b7a3b-03f1-4799-a283-235314973feb-kube-api-access-2c574" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703110 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0dc65c0-bcea-4495-bf55-5095632b71b1" volumeName="kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703130 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0dc65c0-bcea-4495-bf55-5095632b71b1" volumeName="kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703143 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01e345d4-1cb8-4e4f-822d-d71b582b53b3" volumeName="kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703161 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01e345d4-1cb8-4e4f-822d-d71b582b53b3" volumeName="kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703175 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d39b02-7a8c-4edc-ae53-3ebd1e494f58" volumeName="kubernetes.io/secret/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-serving-cert" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703187 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="375eaed5-066a-4615-bc60-09e343aec21b" volumeName="kubernetes.io/secret/375eaed5-066a-4615-bc60-09e343aec21b-serving-cert" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703204 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44c8b7d7-71ad-47ee-9aad-be7f366974f5" volumeName="kubernetes.io/projected/44c8b7d7-71ad-47ee-9aad-be7f366974f5-kube-api-access-987t6" seLinuxMountContext="" Nov 03 22:12:53.703544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703217 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60151974-97b1-4371-9831-762b096bc3fb" volumeName="kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-tmp" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.703234 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fc6de3b-56b8-4344-b7ab-3ad21f645803" volumeName="kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-tmp" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704223 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9994b27-e2c7-4e4a-ab0c-796c19becb5c" volumeName="kubernetes.io/configmap/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-nginx-conf" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704239 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac77e3c-256e-4a95-8106-de9bda8864c5" volumeName="kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704257 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="677c391f-bc16-46c8-9439-02a1f3bf934d" volumeName="kubernetes.io/empty-dir/677c391f-bc16-46c8-9439-02a1f3bf934d-tmp-dir" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704269 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fc6de3b-56b8-4344-b7ab-3ad21f645803" volumeName="kubernetes.io/projected/8fc6de3b-56b8-4344-b7ab-3ad21f645803-kube-api-access-v9lch" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704328 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdd2ca77-cb6a-4b10-906c-841c51aea0c6" volumeName="kubernetes.io/configmap/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-config" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704362 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01e345d4-1cb8-4e4f-822d-d71b582b53b3" volumeName="kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704373 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" volumeName="kubernetes.io/projected/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-kube-api-access-8xlx2" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704401 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60151974-97b1-4371-9831-762b096bc3fb" volumeName="kubernetes.io/projected/60151974-97b1-4371-9831-762b096bc3fb-kube-api-access-bz9qh" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704407 4903 factory.go:55] Registering systemd factory Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704423 4903 factory.go:223] Registration of the systemd container factory successfully Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704410 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2dea624-bc31-43e7-8691-00c686c3eacc" volumeName="kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-cni-binary-copy" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704504 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a714258f-ba4e-4792-853b-7d20411d3ca7" volumeName="kubernetes.io/projected/a714258f-ba4e-4792-853b-7d20411d3ca7-kube-api-access-62rmb" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704518 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1b7053e-001b-4d96-9bf3-1a768e364346" volumeName="kubernetes.io/projected/c1b7053e-001b-4d96-9bf3-1a768e364346-kube-api-access-9t9hd" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704536 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0dc65c0-bcea-4495-bf55-5095632b71b1" volumeName="kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704549 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="375eaed5-066a-4615-bc60-09e343aec21b" volumeName="kubernetes.io/configmap/375eaed5-066a-4615-bc60-09e343aec21b-config" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704588 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44c8b7d7-71ad-47ee-9aad-be7f366974f5" volumeName="kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-env-overrides" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704603 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60151974-97b1-4371-9831-762b096bc3fb" volumeName="kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-service-ca-bundle" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704616 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60151974-97b1-4371-9831-762b096bc3fb" volumeName="kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-trusted-ca-bundle" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704633 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fc6de3b-56b8-4344-b7ab-3ad21f645803" volumeName="kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-tuned" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704645 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1b7053e-001b-4d96-9bf3-1a768e364346" volumeName="kubernetes.io/configmap/c1b7053e-001b-4d96-9bf3-1a768e364346-serviceca" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704662 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0dc65c0-bcea-4495-bf55-5095632b71b1" volumeName="kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates" seLinuxMountContext="" Nov 03 22:12:53.704665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704676 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60151974-97b1-4371-9831-762b096bc3fb" volumeName="kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-snapshots" seLinuxMountContext="" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704692 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdd2ca77-cb6a-4b10-906c-841c51aea0c6" volumeName="kubernetes.io/projected/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-kube-api-access-rzpjd" seLinuxMountContext="" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704710 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44c8b7d7-71ad-47ee-9aad-be7f366974f5" volumeName="kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-config" seLinuxMountContext="" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704727 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="677c391f-bc16-46c8-9439-02a1f3bf934d" volumeName="kubernetes.io/projected/677c391f-bc16-46c8-9439-02a1f3bf934d-kube-api-access-dmrwn" seLinuxMountContext="" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704743 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcefadd9-c607-447d-85ce-fe31bc85e875" volumeName="kubernetes.io/projected/bcefadd9-c607-447d-85ce-fe31bc85e875-kube-api-access-27xh4" seLinuxMountContext="" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704754 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e881588e-ae98-4f04-947d-1a765b8a4e5e" volumeName="kubernetes.io/configmap/e881588e-ae98-4f04-947d-1a765b8a4e5e-konnectivity-ca" seLinuxMountContext="" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704767 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ac77e3c-256e-4a95-8106-de9bda8864c5" volumeName="kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-binary-copy" seLinuxMountContext="" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704787 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63867b60-8f1a-44a6-bbf1-8e521d232081" volumeName="kubernetes.io/projected/63867b60-8f1a-44a6-bbf1-8e521d232081-kube-api-access-vmw5v" seLinuxMountContext="" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704800 4903 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0dc65c0-bcea-4495-bf55-5095632b71b1" volumeName="kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets" seLinuxMountContext="" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704812 4903 reconstruct.go:97] "Volume reconstruction finished" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704813 4903 factory.go:153] Registering CRI-O factory Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704832 4903 factory.go:223] Registration of the crio container factory successfully Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704899 4903 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704820 4903 reconciler.go:29] "Reconciler: start to sync state" Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704917 4903 factory.go:103] Registering Raw factory Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.704939 4903 manager.go:1199] Started watching for new ooms in manager Nov 03 22:12:53.705713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.705406 4903 manager.go:319] Starting recovery of all containers Nov 03 22:12:53.715355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.713331 4903 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 22:12:53.715920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.715894 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:12:53.717984 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.717966 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:12:53.726906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.724581 4903 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 22:12:53.726906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.724617 4903 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 22:12:53.726906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.724815 4903 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 22:12:53.726906 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:53.724876 4903 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 22:12:53.732308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.731720 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:12:53.733987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.733959 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:12:53.739986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.739966 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:12:53.740372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.740343 4903 generic.go:356] "Generic (PLEG): container finished" podID="44c8b7d7-71ad-47ee-9aad-be7f366974f5" containerID="82ec8f71575887f024b0480ab34d429de4998e60223323787289fc2068e51eab" exitCode=1 Nov 03 22:12:53.745789 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.745766 4903 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:53.745947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.745927 4903 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 22:12:53.746402 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.746380 4903 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:53.757081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.757059 4903 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:53.757184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.757120 4903 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 22:12:53.758818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.758794 4903 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="479688a43b62224c324f432951ad096325f22fe1e00d98df3ffa359071585e73" exitCode=0 Nov 03 22:12:53.758818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.758816 4903 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="b36b9a12c1f54bd4c7b97442a1cba64f08b722a53bd7a1d6ad151b7caf30aa1b" exitCode=0 Nov 03 22:12:53.758957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.758823 4903 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="967e2abe1809e6ed13335f6c11ca3e0e5a0d6c7d1f31344b7ce3fb7c7929ba8e" exitCode=0 Nov 03 22:12:53.758957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.758829 4903 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="221456ff2100528f163c05009c2a1671434bef4b0c051f7ea44ed1f6c0b71ddc" exitCode=0 Nov 03 22:12:53.758957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.758834 4903 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="d502b6d2b231ed514545a3b283416145b3f16447087fc75eae34cbec1682477c" exitCode=0 Nov 03 22:12:53.758957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.758840 4903 generic.go:356] "Generic (PLEG): container finished" podID="2ac77e3c-256e-4a95-8106-de9bda8864c5" containerID="88d936261865f83acfca416fadcd5ba2512fdebde0febf07d11b9d260c75fbc8" exitCode=0 Nov 03 22:12:53.764500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.764481 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:12:53.764581 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.764511 4903 generic.go:356] "Generic (PLEG): container finished" podID="0c3d06f5-c335-4cf2-a901-cae1f6c2e142" containerID="0a268de110e22153c69a83cfaf9d6fed6ce349dd349ecd27279f24377dc95850" exitCode=255 Nov 03 22:12:53.770258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.770237 4903 generic.go:356] "Generic (PLEG): container finished" podID="fed587645896ffb2ebb7f46f234764d4" containerID="8d8a2255e7c00795dbf578ba6ab3056676898a73cd66483b6559dc8d65bd706c" exitCode=0 Nov 03 22:12:53.791589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.791518 4903 manager.go:324] Recovery completed Nov 03 22:12:53.811862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.811837 4903 cpu_manager.go:226] "Starting" policy="none" Nov 03 22:12:53.811862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.811857 4903 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 22:12:53.812054 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.811879 4903 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 22:12:53.812114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812047 4903 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 22:12:53.812114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812065 4903 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 22:12:53.812114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812088 4903 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 22:12:53.812114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812097 4903 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 22:12:53.812114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812105 4903 policy_none.go:50] "Start" Nov 03 22:12:53.812366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812117 4903 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 22:12:53.812366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812130 4903 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 22:12:53.812366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812339 4903 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 22:12:53.812366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812351 4903 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 22:12:53.812366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.812359 4903 policy_none.go:44] "Start" Nov 03 22:12:53.816536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.816505 4903 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 22:12:53.820898 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.820881 4903 manager.go:341] "Starting Device Plugin manager" Nov 03 22:12:53.821343 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:53.820906 4903 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 22:12:53.821343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.821343 4903 server.go:87] "Starting device plugin registration server" Nov 03 22:12:53.821592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.821581 4903 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 22:12:53.821663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.821592 4903 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 22:12:53.821735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.821715 4903 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 22:12:53.821974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.821937 4903 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 22:12:53.822043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.821985 4903 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 22:12:53.822043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.821992 4903 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 22:12:53.822161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.822070 4903 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T22:12:53.821971541Z","UUID":"8697302d-b1f6-4144-90d2-2739014230ed","Handler":null,"Name":"","Endpoint":""} Nov 03 22:12:53.823213 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:53.823188 4903 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 22:12:53.823423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.823408 4903 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 22:12:53.823470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.823443 4903 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 22:12:53.825006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.824975 4903 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal"] Nov 03 22:12:53.825687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.825654 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" event={"ID":"fed587645896ffb2ebb7f46f234764d4","Type":"ContainerStarted","Data":"c3470596bd9ad98450e02417e42c2958225ff8cb0e89650ad5adec23afe65824"} Nov 03 22:12:53.825764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.825696 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" event={"ID":"fed587645896ffb2ebb7f46f234764d4","Type":"ContainerDied","Data":"8d8a2255e7c00795dbf578ba6ab3056676898a73cd66483b6559dc8d65bd706c"} Nov 03 22:12:53.825764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.825718 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" event={"ID":"fed587645896ffb2ebb7f46f234764d4","Type":"ContainerStarted","Data":"d50feafba7e23b2aa8fb541005ebc7acaa6448643b3bf15b9843323258174be9"} Nov 03 22:12:53.825764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.825735 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" event={"ID":"ebb54ef3d211c3a17d214f74e032fea7","Type":"ContainerStarted","Data":"78966bec0dd507429c9b3356ffce8d331884270058b776ca5d4ed8bfbdaf024b"} Nov 03 22:12:53.825764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.825748 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" event={"ID":"ebb54ef3d211c3a17d214f74e032fea7","Type":"ContainerStarted","Data":"922f69908bf42654c7cff9ac12ab13fc008bd97a8480ad6253c5e970184b5263"} Nov 03 22:12:53.825977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.825870 4903 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.825977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.825891 4903 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.850758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.850695 4903 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:53.851049 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:53.851030 4903 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-193.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.851124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.850711 4903 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:53.851124 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:53.851106 4903 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.921916 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.921882 4903 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 22:12:53.923234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.923213 4903 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientMemory" Nov 03 22:12:53.923362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.923247 4903 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 22:12:53.923362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.923261 4903 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-193.ec2.internal" event="NodeHasSufficientPID" Nov 03 22:12:53.923362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.923338 4903 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.947047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.947018 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.947144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.947053 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebb54ef3d211c3a17d214f74e032fea7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.ec2.internal\" (UID: \"ebb54ef3d211c3a17d214f74e032fea7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.947144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.947078 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.954363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.954348 4903 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-193.ec2.internal" Nov 03 22:12:53.954418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:53.954406 4903 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.047600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.047519 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.047600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.047559 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.047600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.047578 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebb54ef3d211c3a17d214f74e032fea7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.ec2.internal\" (UID: \"ebb54ef3d211c3a17d214f74e032fea7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.047816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.047628 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.047816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.047645 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fed587645896ffb2ebb7f46f234764d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" (UID: \"fed587645896ffb2ebb7f46f234764d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.047816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.047697 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ebb54ef3d211c3a17d214f74e032fea7-config\") pod \"kube-apiserver-proxy-ip-10-0-133-193.ec2.internal\" (UID: \"ebb54ef3d211c3a17d214f74e032fea7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.685650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.685613 4903 apiserver.go:52] "Watching apiserver" Nov 03 22:12:54.760792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.760723 4903 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 22:12:54.761387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.761360 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2","openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc","kube-system/konnectivity-agent-d4tnh","openshift-ingress-canary/ingress-canary-vpk4w","openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d","openshift-dns/node-resolver-7b4rf","openshift-ingress/router-default-68496b87c-6xdvq","openshift-image-registry/image-registry-655957577d-lgbk6","openshift-dns/dns-default-8vfzp","openshift-insights/insights-runtime-extractor-7vs67","openshift-multus/multus-additional-cni-plugins-nrm7n","kube-system/kas-connection-checker-685895c7c9-grdpd","openshift-ovn-kubernetes/ovnkube-node-djcf8","kube-system/kas-connection-checker-685895c7c9-t67lk","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv","kube-system/global-pull-secret-syncer-vk57x","openshift-multus/multus-pjwnk","kube-system/kas-connection-checker-685895c7c9-l44xs","openshift-multus/network-metrics-daemon-b27xr","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2","kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal","openshift-image-registry/image-registry-5fd57577d7-9mscc","openshift-network-operator/iptables-alerter-hmmwm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5","openshift-network-diagnostics/network-check-target-g7xxx","openshift-console-operator/console-operator-7cd56b8dbc-xsr82","openshift-insights/insights-operator-7c4986d449-xcmlx","openshift-network-console/networking-console-plugin-86c478bb88-zrq7h","openshift-image-registry/node-ca-ctz65","openshift-cluster-node-tuning-operator/tuned-lnbkf"] Nov 03 22:12:54.762906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.762881 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:54.763857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.763838 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.764630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.764609 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:54.765938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.765921 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.766369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.766353 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:54.772390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.772371 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:54.774069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.774051 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.774900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.774871 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.774988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.774964 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.776464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.776444 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:54.776873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.776854 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.776946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.776912 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 22:12:54.777318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777283 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.777564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777473 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 22:12:54.777564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777496 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.777564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777508 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.777564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777517 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 22:12:54.777564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777524 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.777564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777565 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 22:12:54.777896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777574 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 22:12:54.777896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777579 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.777896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777589 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 22:12:54.777896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777628 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 22:12:54.777896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777651 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d79qx\"" Nov 03 22:12:54.777896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777658 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.777896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777715 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h4rt7\"" Nov 03 22:12:54.777896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.777762 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-dg7lc\"" Nov 03 22:12:54.778140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.778047 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vn97d\"" Nov 03 22:12:54.779937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.779920 4903 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.780078 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.780066 4903 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.798936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.798917 4903 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 22:12:54.801574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.801561 4903 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:54.801618 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.801600 4903 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-193.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.810432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.810413 4903 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 22:12:54.810528 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.810457 4903 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-193.ec2.internal" Nov 03 22:12:54.837238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.837215 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.837397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.837275 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 22:12:54.840348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.840331 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.840348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.840338 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.840662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.840645 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.840755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.840744 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 22:12:54.840800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.840754 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.840800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.840779 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.840800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.840781 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.841134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841119 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.841545 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841527 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 22:12:54.841545 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841538 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.841648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841555 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.841648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841563 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 22:12:54.841648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841578 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 22:12:54.841648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841584 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 22:12:54.841648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841594 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.841648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841596 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.841848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841668 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 22:12:54.841848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841712 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 22:12:54.841848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841717 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.841848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841740 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.841954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841857 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.841954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841865 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.842008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841970 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.842008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.841996 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 22:12:54.842239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.842227 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 22:12:54.842365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.842351 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.842398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.842385 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.842511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.842501 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 22:12:54.842680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.842669 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.842731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.842721 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 22:12:54.843132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.843120 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 22:12:54.843175 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.843164 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 22:12:54.843410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.843397 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.843621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.843609 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 22:12:54.843985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.843974 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.845160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.845146 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.845284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.845270 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.845461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.845449 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.845511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.845501 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 22:12:54.845552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.845540 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:12:54.846002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.845983 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.847581 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.847562 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 22:12:54.851499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851482 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-conf\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.851583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851512 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-default-certificate\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.851583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851531 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx28g\" (UniqueName: \"kubernetes.io/projected/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-kube-api-access-cx28g\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:54.851583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851554 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.851677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851600 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.851712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851678 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.851783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851765 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwmt\" (UniqueName: \"kubernetes.io/projected/0bdc4254-64f5-4613-b007-85bc708b2d9b-kube-api-access-2nwmt\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:54.851821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851801 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:54.851856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851831 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-system-cni-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.851888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851859 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.851888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851881 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.851975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851887 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9hd\" (UniqueName: \"kubernetes.io/projected/c1b7053e-001b-4d96-9bf3-1a768e364346-kube-api-access-9t9hd\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:54.851975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851912 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5223b025-80ca-4412-99fb-eed48e128397-data-volume\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.851975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851953 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:54.852114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.851975 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-netns\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.852114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852018 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-hostroot\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.852114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852028 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5223b025-80ca-4412-99fb-eed48e128397-data-volume\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.852114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852056 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:54.852114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852073 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-config\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.852114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852095 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrwn\" (UniqueName: \"kubernetes.io/projected/677c391f-bc16-46c8-9439-02a1f3bf934d-kube-api-access-dmrwn\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:54.852421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852123 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.852421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852144 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cqs8\" (UniqueName: \"kubernetes.io/projected/7c35b7b2-f209-4ff3-ab43-d8533fcdf341-kube-api-access-6cqs8\") pod \"kas-connection-checker-685895c7c9-t67lk\" (UID: \"7c35b7b2-f209-4ff3-ab43-d8533fcdf341\") " pod="kube-system/kas-connection-checker-685895c7c9-t67lk" Nov 03 22:12:54.852421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852191 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.852421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852232 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.852421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852255 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-config\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.852421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852311 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bdc4254-64f5-4613-b007-85bc708b2d9b-cert\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:54.852694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852423 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/375eaed5-066a-4615-bc60-09e343aec21b-serving-cert\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:54.852694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852479 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987t6\" (UniqueName: \"kubernetes.io/projected/44c8b7d7-71ad-47ee-9aad-be7f366974f5-kube-api-access-987t6\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.852694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852502 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9lch\" (UniqueName: \"kubernetes.io/projected/8fc6de3b-56b8-4344-b7ab-3ad21f645803-kube-api-access-v9lch\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.852694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852527 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:54.852694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852569 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/677c391f-bc16-46c8-9439-02a1f3bf934d-hosts-file\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:54.852694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852591 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-k8s-cni-cncf-io\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.852694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852623 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.852694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852667 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-etc-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.853028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852719 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-etc-kubernetes\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.853028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852773 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-sys-fs\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.853028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852811 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:54.853028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852838 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.853028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852885 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-node-log\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.853028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.852927 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-systemd\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.853028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853010 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-service-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.853283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853033 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/375eaed5-066a-4615-bc60-09e343aec21b-config\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:54.853283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853043 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/375eaed5-066a-4615-bc60-09e343aec21b-serving-cert\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:54.853283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853070 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-netns\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.853283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853128 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cf0d1c58-9465-423d-b479-50956f62c7f2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:54.853283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853257 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-snapshots\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853287 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-kubelet\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853336 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-daemon-config\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853286 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/375eaed5-066a-4615-bc60-09e343aec21b-config\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853348 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-snapshots\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853392 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xlx2\" (UniqueName: \"kubernetes.io/projected/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-kube-api-access-8xlx2\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853424 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-tmp-dir\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853446 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-socket-dir-parent\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853478 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqjf\" (UniqueName: \"kubernetes.io/projected/5223b025-80ca-4412-99fb-eed48e128397-kube-api-access-shqjf\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853494 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cf0d1c58-9465-423d-b479-50956f62c7f2-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853531 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.853539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853534 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-tmp-dir\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:54.853980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853566 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxxf\" (UniqueName: \"kubernetes.io/projected/0a18566a-0818-494c-989b-da2abea7edd9-kube-api-access-pvxxf\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.853980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853614 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-config-volume\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:54.853980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853669 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4nt\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.853980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853870 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpjd\" (UniqueName: \"kubernetes.io/projected/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-kube-api-access-rzpjd\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:54.853980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853932 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-daemon-config\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.853980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853930 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.853980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853958 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-config-volume\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:54.854256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.853999 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.854256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854043 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677c391f-bc16-46c8-9439-02a1f3bf934d-tmp-dir\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:54.854256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854107 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677c391f-bc16-46c8-9439-02a1f3bf934d-tmp-dir\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:54.854256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854115 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 22:12:54.854256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854118 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.854256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854163 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-multus-certs\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.854256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854194 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-socket-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.854256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854224 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovn-node-metrics-cert\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.854623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854285 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-run\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.854623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854394 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmw5v\" (UniqueName: \"kubernetes.io/projected/63867b60-8f1a-44a6-bbf1-8e521d232081-kube-api-access-vmw5v\") pod \"kas-connection-checker-685895c7c9-grdpd\" (UID: \"63867b60-8f1a-44a6-bbf1-8e521d232081\") " pod="kube-system/kas-connection-checker-685895c7c9-grdpd" Nov 03 22:12:54.854623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854445 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e881588e-ae98-4f04-947d-1a765b8a4e5e-agent-certs\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:54.854623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854482 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-bin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.854623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854512 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-host\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.854818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854643 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:54.854818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854692 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.854818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854716 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hmn\" (UniqueName: \"kubernetes.io/projected/2ac77e3c-256e-4a95-8106-de9bda8864c5-kube-api-access-j8hmn\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.854818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854746 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-sys\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.854818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854811 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-tmp\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.855006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854944 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:54.855006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854960 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.855006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854975 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60151974-97b1-4371-9831-762b096bc3fb-tmp\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.855006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.854994 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27xh4\" (UniqueName: \"kubernetes.io/projected/bcefadd9-c607-447d-85ce-fe31bc85e875-kube-api-access-27xh4\") pod \"network-check-source-6f78d59f99-q9cfc\" (UID: \"bcefadd9-c607-447d-85ce-fe31bc85e875\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" Nov 03 22:12:54.855155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855052 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftn6\" (UniqueName: \"kubernetes.io/projected/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-kube-api-access-5ftn6\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:54.855155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855109 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.855155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855145 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysconfig\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.855315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855172 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c574\" (UniqueName: \"kubernetes.io/projected/805b7a3b-03f1-4799-a283-235314973feb-kube-api-access-2c574\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:54.855315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855214 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-registration-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.855315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855278 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-var-lib-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.855467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855380 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.855467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855410 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/805b7a3b-03f1-4799-a283-235314973feb-host-slash\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:54.855467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855446 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.855587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855478 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.855587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855540 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1b7053e-001b-4d96-9bf3-1a768e364346-serviceca\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:54.855644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855604 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-cni-binary-copy\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.855644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855636 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-systemd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.855710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855690 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-log-socket\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.855781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855763 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-modprobe-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.855833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855806 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9qh\" (UniqueName: \"kubernetes.io/projected/60151974-97b1-4371-9831-762b096bc3fb-kube-api-access-bz9qh\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.855875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855835 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.855920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855879 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-stats-auth\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.855920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855896 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-trusted-ca\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:54.856014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855929 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2dea624-bc31-43e7-8691-00c686c3eacc-cni-binary-copy\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.856014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.855986 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1b7053e-001b-4d96-9bf3-1a768e364346-serviceca\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:54.856115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856034 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-systemd-units\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.856115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856084 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-tmp\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.856212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856124 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.856212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856153 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.856212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856168 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-tmp\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.856212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856189 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-config\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:54.856417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856274 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.856417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856328 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:54.856417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856355 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25gv\" (UniqueName: \"kubernetes.io/projected/375eaed5-066a-4615-bc60-09e343aec21b-kube-api-access-b25gv\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:54.856417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856383 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.856417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856393 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-lib-modules\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.856417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856409 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 22:12:54.856666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856444 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-config\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:54.856666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856506 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.856666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856513 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60151974-97b1-4371-9831-762b096bc3fb-serving-cert\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.856666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856617 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck4pl\" (UniqueName: \"kubernetes.io/projected/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-kube-api-access-ck4pl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:54.856666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856635 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-conf-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.856666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856651 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5223b025-80ca-4412-99fb-eed48e128397-crio-socket\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.856905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856707 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62rmb\" (UniqueName: \"kubernetes.io/projected/a714258f-ba4e-4792-853b-7d20411d3ca7-kube-api-access-62rmb\") pod \"kas-connection-checker-685895c7c9-l44xs\" (UID: \"a714258f-ba4e-4792-853b-7d20411d3ca7\") " pod="kube-system/kas-connection-checker-685895c7c9-l44xs" Nov 03 22:12:54.856905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856751 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:54.856905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856798 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1b7053e-001b-4d96-9bf3-1a768e364346-host\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:54.856905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856849 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/cf0d1c58-9465-423d-b479-50956f62c7f2-kube-api-access-54ggj\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856918 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856961 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60151974-97b1-4371-9831-762b096bc3fb-serving-cert\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.856968 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857003 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857019 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e881588e-ae98-4f04-947d-1a765b8a4e5e-konnectivity-ca\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857040 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759px\" (UniqueName: \"kubernetes.io/projected/a2dea624-bc31-43e7-8691-00c686c3eacc-kube-api-access-759px\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857057 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-device-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857064 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857075 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:54.857126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857097 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-kubelet\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857153 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857185 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60151974-97b1-4371-9831-762b096bc3fb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857187 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-bin\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857200 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857260 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-env-overrides\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857280 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857287 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-kubernetes\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857342 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-var-lib-kubelet\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857345 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e881588e-ae98-4f04-947d-1a765b8a4e5e-konnectivity-ca\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857371 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpl5\" (UniqueName: \"kubernetes.io/projected/305d6b5c-15d1-4c7e-8b31-db2c71402cd3-kube-api-access-fzpl5\") pod \"volume-data-source-validator-58c9f5f46d-fgqk2\" (UID: \"305d6b5c-15d1-4c7e-8b31-db2c71402cd3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857402 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-kubelet-config\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857419 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-cnibin\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857436 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-serving-cert\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857441 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-env-overrides\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857452 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-netd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.857495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857482 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-dbus\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857548 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857584 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857679 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-multus\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857699 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-tuned\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857714 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857722 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857775 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857782 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-tuned\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857794 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/805b7a3b-03f1-4799-a283-235314973feb-iptables-alerter-script\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857838 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857853 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857868 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-system-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.857903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857900 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.857926 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhtc\" (UniqueName: \"kubernetes.io/projected/bf8fa970-77b7-4f41-9a2d-a586343c51da-kube-api-access-7rhtc\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858000 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-slash\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858008 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858024 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858030 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/805b7a3b-03f1-4799-a283-235314973feb-iptables-alerter-script\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858031 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ac77e3c-256e-4a95-8106-de9bda8864c5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858063 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-ovn\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858085 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858122 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-cnibin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858168 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskmm\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.858351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858194 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-script-lib\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.858667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.858443 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovnkube-script-lib\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.861032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.861018 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 22:12:54.862774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.862758 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 22:12:54.866732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.866715 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-trusted-ca\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:54.895595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.895563 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 22:12:54.895719 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.895611 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 22:12:54.895719 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.895644 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 22:12:54.896141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.896127 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 22:12:54.896321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.896307 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 22:12:54.896375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.896317 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 22:12:54.896375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.896353 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 22:12:54.897755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.897741 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wwhr4\"" Nov 03 22:12:54.897938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.897915 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 22:12:54.898001 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.897952 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 22:12:54.898051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898007 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7shc8\"" Nov 03 22:12:54.898051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898009 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz46p\"" Nov 03 22:12:54.898051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898033 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4tpvw\"" Nov 03 22:12:54.898179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898058 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7gr4r\"" Nov 03 22:12:54.898179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898082 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wf9zk\"" Nov 03 22:12:54.898179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898132 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 22:12:54.898335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898281 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.898335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898321 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 22:12:54.898450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898370 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h4jdd\"" Nov 03 22:12:54.898518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898502 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9wkj9\"" Nov 03 22:12:54.898588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898516 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 22:12:54.898620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898602 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 22:12:54.898662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898648 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 22:12:54.898814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898800 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zwzw5\"" Nov 03 22:12:54.898881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.898862 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l7j9d\"" Nov 03 22:12:54.902861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.902841 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-nt8s4\"" Nov 03 22:12:54.902997 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.902881 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wvrhf\"" Nov 03 22:12:54.902997 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.902943 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.902997 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.902954 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bdc4254-64f5-4613-b007-85bc708b2d9b-cert\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:54.902997 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.902956 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-default-certificate\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.903278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903029 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wwdkz\"" Nov 03 22:12:54.903278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903036 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mdchn\"" Nov 03 22:12:54.903278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903101 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 22:12:54.903278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903126 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 22:12:54.903278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903186 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xf9z9\"" Nov 03 22:12:54.903278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903188 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 22:12:54.903278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903281 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-g7tv5\"" Nov 03 22:12:54.903694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903409 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zdncz\"" Nov 03 22:12:54.903694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903413 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-794rw\"" Nov 03 22:12:54.903694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903491 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mq5fv\"" Nov 03 22:12:54.903694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903511 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h6vzq\"" Nov 03 22:12:54.903694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903516 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jwdqh\"" Nov 03 22:12:54.903694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.903617 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svbdj\"" Nov 03 22:12:54.904262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.904243 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrwn\" (UniqueName: \"kubernetes.io/projected/677c391f-bc16-46c8-9439-02a1f3bf934d-kube-api-access-dmrwn\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:54.904471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.904454 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwmt\" (UniqueName: \"kubernetes.io/projected/0bdc4254-64f5-4613-b007-85bc708b2d9b-kube-api-access-2nwmt\") pod \"ingress-canary-vpk4w\" (UID: \"0bdc4254-64f5-4613-b007-85bc708b2d9b\") " pod="openshift-ingress-canary/ingress-canary-vpk4w" Nov 03 22:12:54.904536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.904495 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9hd\" (UniqueName: \"kubernetes.io/projected/c1b7053e-001b-4d96-9bf3-1a768e364346-kube-api-access-9t9hd\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:54.904584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.904532 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9lch\" (UniqueName: \"kubernetes.io/projected/8fc6de3b-56b8-4344-b7ab-3ad21f645803-kube-api-access-v9lch\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.904584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.904554 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44c8b7d7-71ad-47ee-9aad-be7f366974f5-ovn-node-metrics-cert\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.904720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.904705 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e881588e-ae98-4f04-947d-1a765b8a4e5e-agent-certs\") pod \"konnectivity-agent-d4tnh\" (UID: \"e881588e-ae98-4f04-947d-1a765b8a4e5e\") " pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:54.905285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.905268 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-original-pull-secret\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:54.905699 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.905680 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpjd\" (UniqueName: \"kubernetes.io/projected/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-kube-api-access-rzpjd\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:54.906216 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.906197 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-stats-auth\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.906500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.906484 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cqs8\" (UniqueName: \"kubernetes.io/projected/7c35b7b2-f209-4ff3-ab43-d8533fcdf341-kube-api-access-6cqs8\") pod \"kas-connection-checker-685895c7c9-t67lk\" (UID: \"7c35b7b2-f209-4ff3-ab43-d8533fcdf341\") " pod="kube-system/kas-connection-checker-685895c7c9-t67lk" Nov 03 22:12:54.908566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.908549 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-serving-cert\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:54.911228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.911210 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqjf\" (UniqueName: \"kubernetes.io/projected/5223b025-80ca-4412-99fb-eed48e128397-kube-api-access-shqjf\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.911482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.911464 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-987t6\" (UniqueName: \"kubernetes.io/projected/44c8b7d7-71ad-47ee-9aad-be7f366974f5-kube-api-access-987t6\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.912053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.912034 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.913177 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.913160 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.914749 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.914732 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx28g\" (UniqueName: \"kubernetes.io/projected/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-kube-api-access-cx28g\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:54.920499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.920481 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-759px\" (UniqueName: \"kubernetes.io/projected/a2dea624-bc31-43e7-8691-00c686c3eacc-kube-api-access-759px\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.920568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.920506 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftn6\" (UniqueName: \"kubernetes.io/projected/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-kube-api-access-5ftn6\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:54.922890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.922870 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpl5\" (UniqueName: \"kubernetes.io/projected/305d6b5c-15d1-4c7e-8b31-db2c71402cd3-kube-api-access-fzpl5\") pod \"volume-data-source-validator-58c9f5f46d-fgqk2\" (UID: \"305d6b5c-15d1-4c7e-8b31-db2c71402cd3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgqk2" Nov 03 22:12:54.923458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.923437 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27xh4\" (UniqueName: \"kubernetes.io/projected/bcefadd9-c607-447d-85ce-fe31bc85e875-kube-api-access-27xh4\") pod \"network-check-source-6f78d59f99-q9cfc\" (UID: \"bcefadd9-c607-447d-85ce-fe31bc85e875\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-q9cfc" Nov 03 22:12:54.923748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.923732 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25gv\" (UniqueName: \"kubernetes.io/projected/375eaed5-066a-4615-bc60-09e343aec21b-kube-api-access-b25gv\") pod \"service-ca-operator-544d749c8c-9vt9d\" (UID: \"375eaed5-066a-4615-bc60-09e343aec21b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" Nov 03 22:12:54.924029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.924006 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c574\" (UniqueName: \"kubernetes.io/projected/805b7a3b-03f1-4799-a283-235314973feb-kube-api-access-2c574\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:54.924110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.924029 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck4pl\" (UniqueName: \"kubernetes.io/projected/33d39b02-7a8c-4edc-ae53-3ebd1e494f58-kube-api-access-ck4pl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r85cv\" (UID: \"33d39b02-7a8c-4edc-ae53-3ebd1e494f58\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" Nov 03 22:12:54.924909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.924891 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xlx2\" (UniqueName: \"kubernetes.io/projected/0c3d06f5-c335-4cf2-a901-cae1f6c2e142-kube-api-access-8xlx2\") pod \"console-operator-7cd56b8dbc-xsr82\" (UID: \"0c3d06f5-c335-4cf2-a901-cae1f6c2e142\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:54.925362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.925343 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54ggj\" (UniqueName: \"kubernetes.io/projected/cf0d1c58-9465-423d-b479-50956f62c7f2-kube-api-access-54ggj\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:54.925530 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.925515 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9qh\" (UniqueName: \"kubernetes.io/projected/60151974-97b1-4371-9831-762b096bc3fb-kube-api-access-bz9qh\") pod \"insights-operator-7c4986d449-xcmlx\" (UID: \"60151974-97b1-4371-9831-762b096bc3fb\") " pod="openshift-insights/insights-operator-7c4986d449-xcmlx" Nov 03 22:12:54.925981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.925962 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.926675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.926659 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62rmb\" (UniqueName: \"kubernetes.io/projected/a714258f-ba4e-4792-853b-7d20411d3ca7-kube-api-access-62rmb\") pod \"kas-connection-checker-685895c7c9-l44xs\" (UID: \"a714258f-ba4e-4792-853b-7d20411d3ca7\") " pod="kube-system/kas-connection-checker-685895c7c9-l44xs" Nov 03 22:12:54.927332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.927316 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxxf\" (UniqueName: \"kubernetes.io/projected/0a18566a-0818-494c-989b-da2abea7edd9-kube-api-access-pvxxf\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.927383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.927367 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskmm\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.928201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.928184 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sp7f\" (UniqueName: \"kubernetes.io/projected/019bf19b-ca0e-4a02-af50-d6450151540e-kube-api-access-6sp7f\") pod \"network-check-target-g7xxx\" (UID: \"019bf19b-ca0e-4a02-af50-d6450151540e\") " pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:54.928394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.928379 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhtc\" (UniqueName: \"kubernetes.io/projected/bf8fa970-77b7-4f41-9a2d-a586343c51da-kube-api-access-7rhtc\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.928843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.928807 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4nt\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.929288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.929272 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.929728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.929713 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmw5v\" (UniqueName: \"kubernetes.io/projected/63867b60-8f1a-44a6-bbf1-8e521d232081-kube-api-access-vmw5v\") pod \"kas-connection-checker-685895c7c9-grdpd\" (UID: \"63867b60-8f1a-44a6-bbf1-8e521d232081\") " pod="kube-system/kas-connection-checker-685895c7c9-grdpd" Nov 03 22:12:54.930808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.930792 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hmn\" (UniqueName: \"kubernetes.io/projected/2ac77e3c-256e-4a95-8106-de9bda8864c5-kube-api-access-j8hmn\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.958760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958741 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-slash\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.958760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958767 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.958909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958782 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-ovn\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.958909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958799 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.958909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958816 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-cnibin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.958909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958841 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-conf\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.958909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958849 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.958909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958865 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-ovn\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.958909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958869 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-system-cni-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.958909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958901 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-slash\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958903 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-system-cni-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958903 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958943 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-cnibin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958965 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958973 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958987 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysctl-conf\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.958989 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-netns\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959016 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-netns\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959014 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-hostroot\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959056 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959087 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.958993 4903 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959096 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-hostroot\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959134 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.959158 4903 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959170 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/677c391f-bc16-46c8-9439-02a1f3bf934d-hosts-file\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.959064 4903 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.959203 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls podName:5223b025-80ca-4412-99fb-eed48e128397 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.459181102 +0000 UTC m=+1.863122074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7vs67" (UID: "5223b025-80ca-4412-99fb-eed48e128397") : secret "insights-runtime-extractor-tls" not found Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959233 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-k8s-cni-cncf-io\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959257 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/677c391f-bc16-46c8-9439-02a1f3bf934d-hosts-file\") pod \"node-resolver-7b4rf\" (UID: \"677c391f-bc16-46c8-9439-02a1f3bf934d\") " pod="openshift-dns/node-resolver-7b4rf" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959263 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.959259 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.959274 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.459257647 +0000 UTC m=+1.863198601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.959326 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.459308056 +0000 UTC m=+1.863249013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : secret "metrics-daemon-secret" not found Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959326 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-etc-selinux\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959358 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-etc-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959390 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-etc-kubernetes\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959385 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-etc-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959326 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-k8s-cni-cncf-io\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959418 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-sys-fs\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959429 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-etc-kubernetes\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959448 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959469 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959470 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-sys-fs\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959483 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-node-log\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959498 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-systemd\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959519 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-netns\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959533 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-node-log\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959543 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-kubelet\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959558 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959573 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-systemd\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959574 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-socket-dir-parent\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959576 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-netns\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959599 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-kubelet\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959604 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959661 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959663 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-socket-dir-parent\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.959675 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.459665095 +0000 UTC m=+1.863606066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : configmap references non-existent config key: service-ca.crt Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959699 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959709 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959721 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-multus-certs\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959746 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959751 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-socket-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959776 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-run-multus-certs\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959790 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-run\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959817 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-bin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959840 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-bin\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959842 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-host\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959841 4903 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959843 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-run\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959868 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-sys\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959906 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959880 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-host\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959820 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-socket-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959926 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysconfig\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.959906 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-sys\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.960008 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.959972 4903 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960044 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-sysconfig\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960072 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-registration-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.960077 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls podName:cf0d1c58-9465-423d-b479-50956f62c7f2 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.460060996 +0000 UTC m=+1.864001949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v9lq6" (UID: "cf0d1c58-9465-423d-b479-50956f62c7f2") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960112 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-var-lib-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960134 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-registration-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960148 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-var-lib-openvswitch\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960135 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960183 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/805b7a3b-03f1-4799-a283-235314973feb-host-slash\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.960204 4903 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960235 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/805b7a3b-03f1-4799-a283-235314973feb-host-slash\") pod \"iptables-alerter-hmmwm\" (UID: \"805b7a3b-03f1-4799-a283-235314973feb\") " pod="openshift-network-operator/iptables-alerter-hmmwm" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960204 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.960241 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.460229428 +0000 UTC m=+1.864170384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : secret "router-metrics-certs-default" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.960256 4903 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.960265 4903 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd57577d7-9mscc: secret "image-registry-tls" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960308 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-systemd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.960327 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls podName:d0dc65c0-bcea-4495-bf55-5095632b71b1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.460316185 +0000 UTC m=+1.864257160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls") pod "image-registry-5fd57577d7-9mscc" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1") : secret "image-registry-tls" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960333 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-run-systemd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960340 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-log-socket\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960360 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-modprobe-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960376 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-systemd-units\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960393 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960406 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-log-socket\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960408 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-lib-modules\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960432 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-systemd-units\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960439 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960461 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-conf-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960454 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-multus-conf-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960509 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-lib-modules\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960497 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5223b025-80ca-4412-99fb-eed48e128397-crio-socket\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960532 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-modprobe-d\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960557 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960585 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5223b025-80ca-4412-99fb-eed48e128397-crio-socket\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960601 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1b7053e-001b-4d96-9bf3-1a768e364346-host\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.960641 4903 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960657 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-device-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.960671 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert podName:c9994b27-e2c7-4e4a-ab0c-796c19becb5c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.460662807 +0000 UTC m=+1.864603765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zrq7h" (UID: "c9994b27-e2c7-4e4a-ab0c-796c19becb5c") : secret "networking-console-plugin-cert" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960689 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-kubelet\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960705 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf8fa970-77b7-4f41-9a2d-a586343c51da-device-dir\") pod \"aws-ebs-csi-driver-node-z8vv5\" (UID: \"bf8fa970-77b7-4f41-9a2d-a586343c51da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z8vv5" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960717 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-kubelet\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960725 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1b7053e-001b-4d96-9bf3-1a768e364346-host\") pod \"node-ca-ctz65\" (UID: \"c1b7053e-001b-4d96-9bf3-1a768e364346\") " pod="openshift-image-registry/node-ca-ctz65" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960763 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960791 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-bin\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960814 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-kubernetes\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960837 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-var-lib-kubelet\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960871 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-kubelet-config\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960889 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-etc-kubernetes\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960902 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-cnibin\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960915 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-bin\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960948 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-netd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960955 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fc6de3b-56b8-4344-b7ab-3ad21f645803-var-lib-kubelet\") pod \"tuned-lnbkf\" (UID: \"8fc6de3b-56b8-4344-b7ab-3ad21f645803\") " pod="openshift-cluster-node-tuning-operator/tuned-lnbkf" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.960986 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-dbus\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961014 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-multus\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961010 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ac77e3c-256e-4a95-8106-de9bda8864c5-cnibin\") pod \"multus-additional-cni-plugins-nrm7n\" (UID: \"2ac77e3c-256e-4a95-8106-de9bda8864c5\") " pod="openshift-multus/multus-additional-cni-plugins-nrm7n" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961035 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-kubelet-config\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961041 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961061 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-host-var-lib-cni-multus\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961067 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44c8b7d7-71ad-47ee-9aad-be7f366974f5-host-cni-netd\") pod \"ovnkube-node-djcf8\" (UID: \"44c8b7d7-71ad-47ee-9aad-be7f366974f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961088 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-system-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961096 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05ab17d1-b5dc-4cb6-bf8e-eb650c724aef-dbus\") pod \"global-pull-secret-syncer-vk57x\" (UID: \"05ab17d1-b5dc-4cb6-bf8e-eb650c724aef\") " pod="kube-system/global-pull-secret-syncer-vk57x" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961127 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2dea624-bc31-43e7-8691-00c686c3eacc-system-cni-dir\") pod \"multus-pjwnk\" (UID: \"a2dea624-bc31-43e7-8691-00c686c3eacc\") " pod="openshift-multus/multus-pjwnk" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.961113 4903 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.961153 4903 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655957577d-lgbk6: secret "image-registry-tls" not found Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.961169 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5223b025-80ca-4412-99fb-eed48e128397-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:54.961637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:54.961192 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls podName:01e345d4-1cb8-4e4f-822d-d71b582b53b3 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:55.461179321 +0000 UTC m=+1.865120288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls") pod "image-registry-655957577d-lgbk6" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3") : secret "image-registry-tls" not found Nov 03 22:12:54.963405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:54.962182 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/fdd2ca77-cb6a-4b10-906c-841c51aea0c6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-hdwl2\" (UID: \"fdd2ca77-cb6a-4b10-906c-841c51aea0c6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:55.063437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.063366 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" Nov 03 22:12:55.064465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.064447 4903 scope.go:122] "RemoveContainer" containerID="0a268de110e22153c69a83cfaf9d6fed6ce349dd349ecd27279f24377dc95850" Nov 03 22:12:55.259549 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.259515 4903 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:12:55.390423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.390349 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2"] Nov 03 22:12:55.442582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.442559 4903 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:12:55.464367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.464341 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:55.464470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.464377 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:55.464470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.464409 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:55.464470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.464443 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:55.464579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.464479 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:55.464579 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464486 4903 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:55.464579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.464524 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:55.464579 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464547 4903 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:55.464579 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464557 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.464534679 +0000 UTC m=+2.868475631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:12:55.464579 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464492 4903 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:12:55.464579 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464577 4903 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:12:55.464579 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464583 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.464572751 +0000 UTC m=+2.868513724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : configmap references non-existent config key: service-ca.crt Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464589 4903 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464599 4903 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd57577d7-9mscc: secret "image-registry-tls" not found Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464602 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls podName:cf0d1c58-9465-423d-b479-50956f62c7f2 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.464594084 +0000 UTC m=+2.868535038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v9lq6" (UID: "cf0d1c58-9465-423d-b479-50956f62c7f2") : secret "cluster-monitoring-operator-tls" not found Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464622 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.464614161 +0000 UTC m=+2.868555115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : secret "metrics-daemon-secret" not found Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464634 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.464629639 +0000 UTC m=+2.868570592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : secret "router-metrics-certs-default" not found Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464646 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls podName:d0dc65c0-bcea-4495-bf55-5095632b71b1 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.46464103 +0000 UTC m=+2.868581983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls") pod "image-registry-5fd57577d7-9mscc" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1") : secret "image-registry-tls" not found Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.464683 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.464720 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.464748 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464792 4903 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464802 4903 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655957577d-lgbk6: secret "image-registry-tls" not found Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464822 4903 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:12:55.464881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464834 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls podName:01e345d4-1cb8-4e4f-822d-d71b582b53b3 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.464820183 +0000 UTC m=+2.868761136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls") pod "image-registry-655957577d-lgbk6" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3") : secret "image-registry-tls" not found Nov 03 22:12:55.465249 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:55.464901 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert podName:c9994b27-e2c7-4e4a-ab0c-796c19becb5c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:56.464885852 +0000 UTC m=+2.868826809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zrq7h" (UID: "c9994b27-e2c7-4e4a-ab0c-796c19becb5c") : secret "networking-console-plugin-cert" not found Nov 03 22:12:55.466871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.466855 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5223b025-80ca-4412-99fb-eed48e128397-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7vs67\" (UID: \"5223b025-80ca-4412-99fb-eed48e128397\") " pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:55.678224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.678192 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7vs67" Nov 03 22:12:55.689210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.689184 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7b4rf_677c391f-bc16-46c8-9439-02a1f3bf934d/dns-node-resolver/0.log" Nov 03 22:12:55.784949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.784915 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" event={"ID":"fdd2ca77-cb6a-4b10-906c-841c51aea0c6","Type":"ContainerStarted","Data":"3146702addfed9796770f795e16d31329203971a3e7927f509742063d1abd4c1"} Nov 03 22:12:55.787554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.787526 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:12:55.787838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.787813 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" event={"ID":"0c3d06f5-c335-4cf2-a901-cae1f6c2e142","Type":"ContainerStarted","Data":"a19fed2f7bc8c9eba0cad83ed02201083fa58582e7696268985a8aefe8a07de2"} Nov 03 22:12:55.986083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.985994 4903 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4tnh_e881588e-ae98-4f04-947d-1a765b8a4e5e/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:55.993195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:55.993165 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7vs67"] Nov 03 22:12:55.999423 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:55.999391 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5223b025_80ca_4412_99fb_eed48e128397.slice/crio-ce573a278dbb5e2cfbf3102e0322d4c018aabdce58de7346ba617d7d4f88b98c WatchSource:0}: Error finding container ce573a278dbb5e2cfbf3102e0322d4c018aabdce58de7346ba617d7d4f88b98c: Status 404 returned error can't find the container with id ce573a278dbb5e2cfbf3102e0322d4c018aabdce58de7346ba617d7d4f88b98c Nov 03 22:12:56.196952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.196895 4903 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4tnh_e881588e-ae98-4f04-947d-1a765b8a4e5e/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:56.354318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.354188 4903 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4tnh_e881588e-ae98-4f04-947d-1a765b8a4e5e/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:56.474062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.474031 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:56.474242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.474077 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:56.474242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.474207 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:56.474242 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:56.474235 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.474208356 +0000 UTC m=+4.878149325 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : configmap references non-existent config key: service-ca.crt Nov 03 22:12:56.474430 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.474282 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:56.474430 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:56.474323 4903 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:12:56.474430 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.474357 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:56.474430 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:56.474384 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.47436839 +0000 UTC m=+4.878309348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : secret "router-metrics-certs-default" not found Nov 03 22:12:56.474430 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:56.474422 4903 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 22:12:56.474676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.474442 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:56.474676 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:56.474454 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert podName:c9994b27-e2c7-4e4a-ab0c-796c19becb5c nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.474444113 +0000 UTC m=+4.878385065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zrq7h" (UID: "c9994b27-e2c7-4e4a-ab0c-796c19becb5c") : secret "networking-console-plugin-cert" not found Nov 03 22:12:56.474676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.474486 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:56.474676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.474510 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:56.474676 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:56.474601 4903 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 22:12:56.474676 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:56.474633 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs podName:92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95 nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.474623463 +0000 UTC m=+4.878564418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs") pod "network-metrics-daemon-b27xr" (UID: "92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95") : secret "metrics-daemon-secret" not found Nov 03 22:12:56.474967 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:56.474684 4903 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:56.474967 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:56.474732 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:12:58.474717544 +0000 UTC m=+4.878658513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:12:56.476870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.476826 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"image-registry-655957577d-lgbk6\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:56.476998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.476826 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf0d1c58-9465-423d-b479-50956f62c7f2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v9lq6\" (UID: \"cf0d1c58-9465-423d-b479-50956f62c7f2\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:56.477730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.477707 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"image-registry-5fd57577d7-9mscc\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:56.516054 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.516024 4903 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4tnh_e881588e-ae98-4f04-947d-1a765b8a4e5e/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:56.565033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.564995 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" Nov 03 22:12:56.567174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.567151 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:12:56.576358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.575909 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:12:56.687578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.687517 4903 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4tnh_e881588e-ae98-4f04-947d-1a765b8a4e5e/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:56.701869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.701843 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctz65_c1b7053e-001b-4d96-9bf3-1a768e364346/node-ca/0.log" Nov 03 22:12:56.792867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.792836 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vs67" event={"ID":"5223b025-80ca-4412-99fb-eed48e128397","Type":"ContainerStarted","Data":"138dae03cf36994ef86baa6cc0f1c5a4394611a7fadaedfa646e56f07e78e612"} Nov 03 22:12:56.792867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.792865 4903 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:12:56.793052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.792902 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vs67" event={"ID":"5223b025-80ca-4412-99fb-eed48e128397","Type":"ContainerStarted","Data":"ce573a278dbb5e2cfbf3102e0322d4c018aabdce58de7346ba617d7d4f88b98c"} Nov 03 22:12:56.890228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.890199 4903 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d4tnh_e881588e-ae98-4f04-947d-1a765b8a4e5e/konnectivity-agent/0.log" limit=1024 Nov 03 22:12:56.893653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.893630 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6"] Nov 03 22:12:56.896919 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:56.896892 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0d1c58_9465_423d_b479_50956f62c7f2.slice/crio-570159292f3ac6e653d2641e046020dd3c27a094069e19db2bf68742670a99f2 WatchSource:0}: Error finding container 570159292f3ac6e653d2641e046020dd3c27a094069e19db2bf68742670a99f2: Status 404 returned error can't find the container with id 570159292f3ac6e653d2641e046020dd3c27a094069e19db2bf68742670a99f2 Nov 03 22:12:56.913312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.913273 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fd57577d7-9mscc"] Nov 03 22:12:56.916271 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:56.916250 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0dc65c0_bcea_4495_bf55_5095632b71b1.slice/crio-aa734418220c34c286aa79d5569883a058f7b47ddc706b869a6bd92d2128b652 WatchSource:0}: Error finding container aa734418220c34c286aa79d5569883a058f7b47ddc706b869a6bd92d2128b652: Status 404 returned error can't find the container with id aa734418220c34c286aa79d5569883a058f7b47ddc706b869a6bd92d2128b652 Nov 03 22:12:56.919484 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:56.919463 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-655957577d-lgbk6"] Nov 03 22:12:56.923336 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:56.923286 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e345d4_1cb8_4e4f_822d_d71b582b53b3.slice/crio-27801c01e29682ae577a7fd08f280c11586079aea1315444e6afc99e1d9e69a1 WatchSource:0}: Error finding container 27801c01e29682ae577a7fd08f280c11586079aea1315444e6afc99e1d9e69a1: Status 404 returned error can't find the container with id 27801c01e29682ae577a7fd08f280c11586079aea1315444e6afc99e1d9e69a1 Nov 03 22:12:57.032423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.031781 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d4tnh" Nov 03 22:12:57.517416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.517383 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:12:57.799393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.799275 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655957577d-lgbk6" event={"ID":"01e345d4-1cb8-4e4f-822d-d71b582b53b3","Type":"ContainerStarted","Data":"845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134"} Nov 03 22:12:57.799393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.799359 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655957577d-lgbk6" event={"ID":"01e345d4-1cb8-4e4f-822d-d71b582b53b3","Type":"ContainerStarted","Data":"27801c01e29682ae577a7fd08f280c11586079aea1315444e6afc99e1d9e69a1"} Nov 03 22:12:57.802206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.802178 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" event={"ID":"fdd2ca77-cb6a-4b10-906c-841c51aea0c6","Type":"ContainerStarted","Data":"11bc97f86150e5274fcf42a885a321b859e825aa206199cfbcdf6634662362e5"} Nov 03 22:12:57.802359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.802230 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-hdwl2" event={"ID":"fdd2ca77-cb6a-4b10-906c-841c51aea0c6","Type":"ContainerStarted","Data":"a80b78de565e0e47304d896745971074c7727b5b0932abebcb44c2f7ab4e3925"} Nov 03 22:12:57.804510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.804435 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" event={"ID":"d0dc65c0-bcea-4495-bf55-5095632b71b1","Type":"ContainerStarted","Data":"afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c"} Nov 03 22:12:57.804510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.804481 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" event={"ID":"d0dc65c0-bcea-4495-bf55-5095632b71b1","Type":"ContainerStarted","Data":"aa734418220c34c286aa79d5569883a058f7b47ddc706b869a6bd92d2128b652"} Nov 03 22:12:57.806614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.806578 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vs67" event={"ID":"5223b025-80ca-4412-99fb-eed48e128397","Type":"ContainerStarted","Data":"2ece9c513bc646c87519604ad10261063aaa312f6b3f83a6c08b16e3424684ce"} Nov 03 22:12:57.808225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.808205 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" event={"ID":"cf0d1c58-9465-423d-b479-50956f62c7f2","Type":"ContainerStarted","Data":"570159292f3ac6e653d2641e046020dd3c27a094069e19db2bf68742670a99f2"} Nov 03 22:12:57.889099 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:57.889033 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vpk4w_0bdc4254-64f5-4613-b007-85bc708b2d9b/serve-healthcheck-canary/0.log" Nov 03 22:12:58.493814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.493788 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-r85cv_33d39b02-7a8c-4edc-ae53-3ebd1e494f58/kube-storage-version-migrator-operator/0.log" Nov 03 22:12:58.497077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.497055 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:58.497161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.497108 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:12:58.497161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.497142 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:58.497252 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:58.497202 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:02.497186129 +0000 UTC m=+8.901127087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : configmap references non-existent config key: service-ca.crt Nov 03 22:12:58.497252 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:58.497226 4903 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 22:12:58.497252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.497244 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:12:58.497436 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:58.497273 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs podName:0a18566a-0818-494c-989b-da2abea7edd9 nodeName:}" failed. No retries permitted until 2026-11-03 22:13:02.497258523 +0000 UTC m=+8.901199501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs") pod "router-default-68496b87c-6xdvq" (UID: "0a18566a-0818-494c-989b-da2abea7edd9") : secret "router-metrics-certs-default" not found Nov 03 22:12:58.497436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.497322 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:58.497436 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:58.497331 4903 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:12:58.497436 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:12:58.497368 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:13:02.497356447 +0000 UTC m=+8.901297415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:12:58.499527 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.499500 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95-metrics-certs\") pod \"network-metrics-daemon-b27xr\" (UID: \"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95\") " pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:58.499849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.499829 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9994b27-e2c7-4e4a-ab0c-796c19becb5c-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zrq7h\" (UID: \"c9994b27-e2c7-4e4a-ab0c-796c19becb5c\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:58.667106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.667077 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" Nov 03 22:12:58.673439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.673414 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b27xr" Nov 03 22:12:58.813361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.813315 4903 generic.go:356] "Generic (PLEG): container finished" podID="60151974-97b1-4371-9831-762b096bc3fb" containerID="d730e0fea10787854789487fd884518ace03e18e449353e4030f0fb28ac10108" exitCode=0 Nov 03 22:12:58.813777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.813389 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xcmlx" event={"ID":"60151974-97b1-4371-9831-762b096bc3fb","Type":"ContainerDied","Data":"d730e0fea10787854789487fd884518ace03e18e449353e4030f0fb28ac10108"} Nov 03 22:12:58.813777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.813768 4903 scope.go:122] "RemoveContainer" containerID="d730e0fea10787854789487fd884518ace03e18e449353e4030f0fb28ac10108" Nov 03 22:12:58.815041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.815012 4903 generic.go:356] "Generic (PLEG): container finished" podID="375eaed5-066a-4615-bc60-09e343aec21b" containerID="8cf712c7d94ef9c861c3a00b3a07c35a25103e7fd76bcaa8c3d9b7a4077c757b" exitCode=0 Nov 03 22:12:58.815161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.815077 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" event={"ID":"375eaed5-066a-4615-bc60-09e343aec21b","Type":"ContainerDied","Data":"8cf712c7d94ef9c861c3a00b3a07c35a25103e7fd76bcaa8c3d9b7a4077c757b"} Nov 03 22:12:58.815466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.815447 4903 scope.go:122] "RemoveContainer" containerID="8cf712c7d94ef9c861c3a00b3a07c35a25103e7fd76bcaa8c3d9b7a4077c757b" Nov 03 22:12:58.817794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.817703 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7vs67" event={"ID":"5223b025-80ca-4412-99fb-eed48e128397","Type":"ContainerStarted","Data":"9e6a5ffffa81b9fce3a24a46d4984608b97f3c7536f2efceb3b18ae9ab4821d8"} Nov 03 22:12:58.819263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.819238 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v9lq6" event={"ID":"cf0d1c58-9465-423d-b479-50956f62c7f2","Type":"ContainerStarted","Data":"b36fa727ae7355e95743c9a5be48c1c1234802d65980bd0d303ab0d7b143160b"} Nov 03 22:12:58.819404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.819279 4903 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:12:58.819404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.819344 4903 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:12:58.983088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.983017 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zrq7h"] Nov 03 22:12:58.986362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:58.986337 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b27xr"] Nov 03 22:12:58.986680 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:58.986654 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9994b27_e2c7_4e4a_ab0c_796c19becb5c.slice/crio-41f3060739ec64f50a066f5e23335a4eab88da8f9e67425a994be3a1cb4f0c7e WatchSource:0}: Error finding container 41f3060739ec64f50a066f5e23335a4eab88da8f9e67425a994be3a1cb4f0c7e: Status 404 returned error can't find the container with id 41f3060739ec64f50a066f5e23335a4eab88da8f9e67425a994be3a1cb4f0c7e Nov 03 22:12:58.994667 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:12:58.994642 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e10542_e0ce_4dd4_bbbd_9e24d2c2ff95.slice/crio-2ba25b67faeb519f7d5dd0ae322ae31231cfa9479cd277610d43857d8ca6cac9 WatchSource:0}: Error finding container 2ba25b67faeb519f7d5dd0ae322ae31231cfa9479cd277610d43857d8ca6cac9: Status 404 returned error can't find the container with id 2ba25b67faeb519f7d5dd0ae322ae31231cfa9479cd277610d43857d8ca6cac9 Nov 03 22:12:59.111321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:59.111266 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:12:59.828050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:59.828005 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-xcmlx" event={"ID":"60151974-97b1-4371-9831-762b096bc3fb","Type":"ContainerStarted","Data":"30be0370ad9dd15f7a5d6c32fa72def842587b0a682a85768d853e1fdd48c19c"} Nov 03 22:12:59.830409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:59.830372 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9vt9d" event={"ID":"375eaed5-066a-4615-bc60-09e343aec21b","Type":"ContainerStarted","Data":"6d01e484cb940102241839a22b1eb24e65916f7b345f6e898ce7e1ad9c8f8ddf"} Nov 03 22:12:59.832146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:59.832111 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b27xr" event={"ID":"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95","Type":"ContainerStarted","Data":"2ba25b67faeb519f7d5dd0ae322ae31231cfa9479cd277610d43857d8ca6cac9"} Nov 03 22:12:59.833761 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:12:59.833688 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" event={"ID":"c9994b27-e2c7-4e4a-ab0c-796c19becb5c","Type":"ContainerStarted","Data":"41f3060739ec64f50a066f5e23335a4eab88da8f9e67425a994be3a1cb4f0c7e"} Nov 03 22:13:00.839240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:00.839196 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b27xr" event={"ID":"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95","Type":"ContainerStarted","Data":"4504d7c11f516995da365cb7b6d7f441004acadba1b6ba2c410ef5c74fbc2366"} Nov 03 22:13:00.839691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:00.839262 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b27xr" event={"ID":"92e10542-e0ce-4dd4-bbbd-9e24d2c2ff95","Type":"ContainerStarted","Data":"1499009aef1b7a37c0a3a4752074a07a823803d63dedd2de0e82ecbcd2e0cf1f"} Nov 03 22:13:00.840710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:00.840688 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zrq7h" event={"ID":"c9994b27-e2c7-4e4a-ab0c-796c19becb5c","Type":"ContainerStarted","Data":"1d096519a1945c5fbe1fc67a7ecc9e437631e342157ac058689aaaa7658e213c"} Nov 03 22:13:00.929110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:00.929079 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:13:02.538218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:02.538178 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:13:02.538640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:02.538256 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:13:02.538640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:02.538288 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:13:02.538640 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:13:02.538415 4903 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 22:13:02.538640 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:13:02.538484 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls podName:e302a1bc-43db-4e8b-8ed4-f42194f5ddeb nodeName:}" failed. No retries permitted until 2026-11-03 22:13:10.53846938 +0000 UTC m=+16.942410347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls") pod "dns-default-8vfzp" (UID: "e302a1bc-43db-4e8b-8ed4-f42194f5ddeb") : secret "dns-default-metrics-tls" not found Nov 03 22:13:02.538881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:02.538863 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a18566a-0818-494c-989b-da2abea7edd9-service-ca-bundle\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:13:02.540713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:02.540685 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a18566a-0818-494c-989b-da2abea7edd9-metrics-certs\") pod \"router-default-68496b87c-6xdvq\" (UID: \"0a18566a-0818-494c-989b-da2abea7edd9\") " pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:13:02.564501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:02.564469 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:13:02.875400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:02.875321 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68496b87c-6xdvq"] Nov 03 22:13:02.878719 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:13:02.878690 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a18566a_0818_494c_989b_da2abea7edd9.slice/crio-0683d7432ab9db8ca85a05a9c40fef5604553f9480271552c8d036aa6cd22f4d WatchSource:0}: Error finding container 0683d7432ab9db8ca85a05a9c40fef5604553f9480271552c8d036aa6cd22f4d: Status 404 returned error can't find the container with id 0683d7432ab9db8ca85a05a9c40fef5604553f9480271552c8d036aa6cd22f4d Nov 03 22:13:02.913962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:02.913936 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:13:03.233230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:03.233189 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:13:03.233404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:03.233348 4903 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:13:03.237607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:03.237579 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-xsr82" Nov 03 22:13:03.852250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:03.852218 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68496b87c-6xdvq" event={"ID":"0a18566a-0818-494c-989b-da2abea7edd9","Type":"ContainerStarted","Data":"f16775df46fb353dd3a03c38f831ea8fb3d59ac545585bee3d93947c6e69252f"} Nov 03 22:13:03.852680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:03.852267 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68496b87c-6xdvq" event={"ID":"0a18566a-0818-494c-989b-da2abea7edd9","Type":"ContainerStarted","Data":"0683d7432ab9db8ca85a05a9c40fef5604553f9480271552c8d036aa6cd22f4d"} Nov 03 22:13:03.981558 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:03.981526 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:13:03.981770 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:03.981759 4903 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:13:04.565371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:04.565335 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:13:04.568008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:04.567981 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:13:05.858838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:05.858805 4903 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:13:07.715894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:07.715859 4903 patch_prober.go:29] interesting pod/image-registry-5fd57577d7-9mscc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 22:13:07.716357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:07.715947 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" podUID="d0dc65c0-bcea-4495-bf55-5095632b71b1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:13:08.455192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:08.455157 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:13:08.455434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:08.455363 4903 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 22:13:08.456456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:08.456433 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68496b87c-6xdvq" Nov 03 22:13:09.457936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:09.457910 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-655957577d-lgbk6"] Nov 03 22:13:10.604079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:10.604048 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:13:10.606344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:10.606327 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e302a1bc-43db-4e8b-8ed4-f42194f5ddeb-metrics-tls\") pod \"dns-default-8vfzp\" (UID: \"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb\") " pod="openshift-dns/dns-default-8vfzp" Nov 03 22:13:10.677651 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:10.677619 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vfzp" Nov 03 22:13:10.969737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:10.969704 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8vfzp"] Nov 03 22:13:10.972676 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:13:10.972647 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode302a1bc_43db_4e8b_8ed4_f42194f5ddeb.slice/crio-7328123b9fcc630a0aba40bcd83b9246c3c9087f315b4a8e2ebaa2bd6825f0ce WatchSource:0}: Error finding container 7328123b9fcc630a0aba40bcd83b9246c3c9087f315b4a8e2ebaa2bd6825f0ce: Status 404 returned error can't find the container with id 7328123b9fcc630a0aba40bcd83b9246c3c9087f315b4a8e2ebaa2bd6825f0ce Nov 03 22:13:11.880157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:11.880115 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8vfzp" event={"ID":"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb","Type":"ContainerStarted","Data":"7328123b9fcc630a0aba40bcd83b9246c3c9087f315b4a8e2ebaa2bd6825f0ce"} Nov 03 22:13:12.497173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:12.497145 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7b4rf_677c391f-bc16-46c8-9439-02a1f3bf934d/dns-node-resolver/0.log" Nov 03 22:13:12.885816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:12.885780 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8vfzp" event={"ID":"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb","Type":"ContainerStarted","Data":"472e47e77529608681b278b10929dce3bd9631a246de1cf14673c0042566ca6c"} Nov 03 22:13:12.886321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:12.885842 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8vfzp" event={"ID":"e302a1bc-43db-4e8b-8ed4-f42194f5ddeb","Type":"ContainerStarted","Data":"f290c71850d30c327ebf2c9ea441ca775ec375b421972dce265096e790e368ea"} Nov 03 22:13:12.886321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:12.885946 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8vfzp" Nov 03 22:13:13.099580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:13.099504 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fd57577d7-9mscc_d0dc65c0-bcea-4495-bf55-5095632b71b1/registry/0.log" Nov 03 22:13:13.296614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:13.296539 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-655957577d-lgbk6_01e345d4-1cb8-4e4f-822d-d71b582b53b3/registry/0.log" Nov 03 22:13:13.695808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:13.695782 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctz65_c1b7053e-001b-4d96-9bf3-1a768e364346/node-ca/0.log" Nov 03 22:13:14.296586 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:14.296557 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68496b87c-6xdvq_0a18566a-0818-494c-989b-da2abea7edd9/router/0.log" Nov 03 22:13:14.894923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:14.894897 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vpk4w_0bdc4254-64f5-4613-b007-85bc708b2d9b/serve-healthcheck-canary/0.log" Nov 03 22:13:16.430165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.430129 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2vzkk"] Nov 03 22:13:16.438580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.438558 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459103 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459076 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzncr\" (UniqueName: \"kubernetes.io/projected/2e77d4d0-4aba-43cc-92df-7f9073838727-kube-api-access-pzncr\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459114 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e77d4d0-4aba-43cc-92df-7f9073838727-metrics-client-ca\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459193 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-textfile\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459218 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e77d4d0-4aba-43cc-92df-7f9073838727-root\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459238 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e77d4d0-4aba-43cc-92df-7f9073838727-sys\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459256 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-tls\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459285 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-accelerators-collector-config\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459362 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-wtmp\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459411 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.459639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459624 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xxldp\"" Nov 03 22:13:16.459678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.459637 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 22:13:16.462391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.462361 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 22:13:16.462391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.462365 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 22:13:16.462561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.462393 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 22:13:16.560408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560371 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e77d4d0-4aba-43cc-92df-7f9073838727-metrics-client-ca\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560411 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-textfile\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560433 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e77d4d0-4aba-43cc-92df-7f9073838727-root\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560454 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e77d4d0-4aba-43cc-92df-7f9073838727-sys\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560481 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-tls\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560496 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e77d4d0-4aba-43cc-92df-7f9073838727-root\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560502 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-accelerators-collector-config\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560553 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-wtmp\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560581 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e77d4d0-4aba-43cc-92df-7f9073838727-sys\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560582 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.560656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560660 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzncr\" (UniqueName: \"kubernetes.io/projected/2e77d4d0-4aba-43cc-92df-7f9073838727-kube-api-access-pzncr\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.561096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560756 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-wtmp\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.561096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.560785 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-textfile\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.561193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.561103 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e77d4d0-4aba-43cc-92df-7f9073838727-metrics-client-ca\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.561193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.561109 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-accelerators-collector-config\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.562928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.562898 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-tls\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.563087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.563070 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e77d4d0-4aba-43cc-92df-7f9073838727-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.581568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.581540 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzncr\" (UniqueName: \"kubernetes.io/projected/2e77d4d0-4aba-43cc-92df-7f9073838727-kube-api-access-pzncr\") pod \"node-exporter-2vzkk\" (UID: \"2e77d4d0-4aba-43cc-92df-7f9073838727\") " pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.764710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.764635 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2vzkk" Nov 03 22:13:16.803487 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:13:16.803456 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e77d4d0_4aba_43cc_92df_7f9073838727.slice/crio-38999707d38f8049305559e6aac968d09c47aa4bdad3a3f0a9470a42aeb792cd WatchSource:0}: Error finding container 38999707d38f8049305559e6aac968d09c47aa4bdad3a3f0a9470a42aeb792cd: Status 404 returned error can't find the container with id 38999707d38f8049305559e6aac968d09c47aa4bdad3a3f0a9470a42aeb792cd Nov 03 22:13:16.901090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:16.901048 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2vzkk" event={"ID":"2e77d4d0-4aba-43cc-92df-7f9073838727","Type":"ContainerStarted","Data":"38999707d38f8049305559e6aac968d09c47aa4bdad3a3f0a9470a42aeb792cd"} Nov 03 22:13:18.381910 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:13:18.381833 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:13:18.881371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:18.881331 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djcf8" Nov 03 22:13:18.910370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:18.910336 4903 generic.go:356] "Generic (PLEG): container finished" podID="2e77d4d0-4aba-43cc-92df-7f9073838727" containerID="9e5ec88f2958c733826f26cd89bc346b6a4b6adc2bde2f935922642d6a080792" exitCode=0 Nov 03 22:13:18.910524 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:18.910387 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2vzkk" event={"ID":"2e77d4d0-4aba-43cc-92df-7f9073838727","Type":"ContainerDied","Data":"9e5ec88f2958c733826f26cd89bc346b6a4b6adc2bde2f935922642d6a080792"} Nov 03 22:13:19.463917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:19.463888 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:13:19.916981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:19.916949 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2vzkk" event={"ID":"2e77d4d0-4aba-43cc-92df-7f9073838727","Type":"ContainerStarted","Data":"ee37b39547daf995ebbf18bd4839be1e1fe5e05e56abd1abbb45c02c7dcf7e49"} Nov 03 22:13:19.917142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:19.917000 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2vzkk" event={"ID":"2e77d4d0-4aba-43cc-92df-7f9073838727","Type":"ContainerStarted","Data":"b7f6290da2cff217ad670e31b40e62cd796638ddfa07aa5f87adae59c66a3168"} Nov 03 22:13:19.958123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:19.958075 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2vzkk" podStartSLOduration=2.576122545 podStartE2EDuration="3.958060777s" totalImagesPullingTime="1.296135548s" totalInitContainerRuntime="85.802684ms" isStatefulPod=true podCreationTimestamp="2026-11-03 22:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:13:19.957912113 +0000 UTC m=+26.361853090" watchObservedRunningTime="2026-11-03 22:13:19.958060777 +0000 UTC m=+26.362001751" Nov 03 22:13:20.933635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:20.933609 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:13:22.891811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:22.891780 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8vfzp" Nov 03 22:13:27.520646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:27.520616 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g7xxx" Nov 03 22:13:32.277096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:32.277062 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fd57577d7-9mscc"] Nov 03 22:13:34.477925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.477871 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-655957577d-lgbk6" podUID="01e345d4-1cb8-4e4f-822d-d71b582b53b3" containerName="registry" containerID="cri-o://845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134" gracePeriod=30 Nov 03 22:13:34.721692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.721665 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:13:34.818369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.818282 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt\") pod \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " Nov 03 22:13:34.818369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.818334 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets\") pod \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " Nov 03 22:13:34.818369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.818367 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") pod \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " Nov 03 22:13:34.818645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.818483 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca\") pod \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " Nov 03 22:13:34.818645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.818551 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration\") pod \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " Nov 03 22:13:34.818645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.818607 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates\") pod \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " Nov 03 22:13:34.818799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.818723 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted\") pod \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " Nov 03 22:13:34.818799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.818776 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token\") pod \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\" (UID: \"01e345d4-1cb8-4e4f-822d-d71b582b53b3\") " Nov 03 22:13:34.818923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.818900 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca" pod "01e345d4-1cb8-4e4f-822d-d71b582b53b3" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:34.819029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.819014 4903 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-trusted-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:34.819093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.819011 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates" pod "01e345d4-1cb8-4e4f-822d-d71b582b53b3" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:34.821255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.821227 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token" pod "01e345d4-1cb8-4e4f-822d-d71b582b53b3" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:34.821424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.821256 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets" pod "01e345d4-1cb8-4e4f-822d-d71b582b53b3" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:34.821424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.821371 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls" pod "01e345d4-1cb8-4e4f-822d-d71b582b53b3" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:34.821424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.821381 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt" pod "01e345d4-1cb8-4e4f-822d-d71b582b53b3" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3"). InnerVolumeSpecName "kube-api-access-jq4nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:34.821424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.821385 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration" pod "01e345d4-1cb8-4e4f-822d-d71b582b53b3" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:34.827627 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.827600 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted" pod "01e345d4-1cb8-4e4f-822d-d71b582b53b3" (UID: "01e345d4-1cb8-4e4f-822d-d71b582b53b3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:13:34.920347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.920313 4903 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-image-registry-private-configuration\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:34.920347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.920340 4903 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-certificates\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:34.920347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.920351 4903 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01e345d4-1cb8-4e4f-822d-d71b582b53b3-ca-trust-extracted\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:34.920563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.920361 4903 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-bound-sa-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:34.920563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.920370 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq4nt\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-kube-api-access-jq4nt\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:34.920563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.920379 4903 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01e345d4-1cb8-4e4f-822d-d71b582b53b3-installation-pull-secrets\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:34.920563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.920388 4903 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01e345d4-1cb8-4e4f-822d-d71b582b53b3-registry-tls\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:34.976220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.975410 4903 generic.go:356] "Generic (PLEG): container finished" podID="01e345d4-1cb8-4e4f-822d-d71b582b53b3" containerID="845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134" exitCode=0 Nov 03 22:13:34.976220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.975534 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655957577d-lgbk6" event={"ID":"01e345d4-1cb8-4e4f-822d-d71b582b53b3","Type":"ContainerDied","Data":"845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134"} Nov 03 22:13:34.976220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.975582 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655957577d-lgbk6" event={"ID":"01e345d4-1cb8-4e4f-822d-d71b582b53b3","Type":"ContainerDied","Data":"27801c01e29682ae577a7fd08f280c11586079aea1315444e6afc99e1d9e69a1"} Nov 03 22:13:34.976220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.975608 4903 scope.go:122] "RemoveContainer" containerID="845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134" Nov 03 22:13:34.976220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.975769 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655957577d-lgbk6" Nov 03 22:13:34.985465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.985448 4903 scope.go:122] "RemoveContainer" containerID="845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134" Nov 03 22:13:34.985711 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:13:34.985689 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134\": container with ID starting with 845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134 not found: ID does not exist" containerID="845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134" Nov 03 22:13:34.985758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:34.985724 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134"} err="failed to get container status \"845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134\": rpc error: code = NotFound desc = could not find container \"845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134\": container with ID starting with 845ea472c47d070a535fedd91cc38230fdce735fb183a885910b91518dfb9134 not found: ID does not exist" Nov 03 22:13:35.020930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:35.020899 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-655957577d-lgbk6"] Nov 03 22:13:35.026995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:35.026975 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-655957577d-lgbk6"] Nov 03 22:13:35.730265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:35.730231 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e345d4-1cb8-4e4f-822d-d71b582b53b3" path="/var/lib/kubelet/pods/01e345d4-1cb8-4e4f-822d-d71b582b53b3/volumes" Nov 03 22:13:48.382512 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:13:48.382474 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:13:57.296714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.296648 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" podUID="d0dc65c0-bcea-4495-bf55-5095632b71b1" containerName="registry" containerID="cri-o://afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c" gracePeriod=30 Nov 03 22:13:57.545489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.545461 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:13:57.699574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.699538 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates\") pod \"d0dc65c0-bcea-4495-bf55-5095632b71b1\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " Nov 03 22:13:57.699765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.699589 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca\") pod \"d0dc65c0-bcea-4495-bf55-5095632b71b1\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " Nov 03 22:13:57.699765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.699629 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted\") pod \"d0dc65c0-bcea-4495-bf55-5095632b71b1\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " Nov 03 22:13:57.699765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.699670 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets\") pod \"d0dc65c0-bcea-4495-bf55-5095632b71b1\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " Nov 03 22:13:57.699765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.699700 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration\") pod \"d0dc65c0-bcea-4495-bf55-5095632b71b1\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " Nov 03 22:13:57.699983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.699836 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token\") pod \"d0dc65c0-bcea-4495-bf55-5095632b71b1\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " Nov 03 22:13:57.699983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.699887 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") pod \"d0dc65c0-bcea-4495-bf55-5095632b71b1\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " Nov 03 22:13:57.699983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.699938 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm\") pod \"d0dc65c0-bcea-4495-bf55-5095632b71b1\" (UID: \"d0dc65c0-bcea-4495-bf55-5095632b71b1\") " Nov 03 22:13:57.700141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.700020 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates" pod "d0dc65c0-bcea-4495-bf55-5095632b71b1" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:57.700200 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.700137 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca" pod "d0dc65c0-bcea-4495-bf55-5095632b71b1" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:13:57.700255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.700210 4903 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-certificates\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:57.702890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.702307 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token" pod "d0dc65c0-bcea-4495-bf55-5095632b71b1" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:57.702890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.702526 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration" pod "d0dc65c0-bcea-4495-bf55-5095632b71b1" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:57.702890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.702534 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls" pod "d0dc65c0-bcea-4495-bf55-5095632b71b1" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:57.702890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.702653 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets" pod "d0dc65c0-bcea-4495-bf55-5095632b71b1" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:13:57.702890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.702678 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm" pod "d0dc65c0-bcea-4495-bf55-5095632b71b1" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1"). InnerVolumeSpecName "kube-api-access-fskmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:13:57.706670 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.706617 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted" pod "d0dc65c0-bcea-4495-bf55-5095632b71b1" (UID: "d0dc65c0-bcea-4495-bf55-5095632b71b1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:13:57.801607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.801574 4903 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-bound-sa-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:57.801607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.801602 4903 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-registry-tls\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:57.801607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.801611 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fskmm\" (UniqueName: \"kubernetes.io/projected/d0dc65c0-bcea-4495-bf55-5095632b71b1-kube-api-access-fskmm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:57.801810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.801623 4903 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0dc65c0-bcea-4495-bf55-5095632b71b1-trusted-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:57.801810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.801634 4903 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0dc65c0-bcea-4495-bf55-5095632b71b1-ca-trust-extracted\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:57.801810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.801648 4903 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-installation-pull-secrets\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:57.801810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:57.801661 4903 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0dc65c0-bcea-4495-bf55-5095632b71b1-image-registry-private-configuration\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:13:58.059163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:58.059081 4903 generic.go:356] "Generic (PLEG): container finished" podID="d0dc65c0-bcea-4495-bf55-5095632b71b1" containerID="afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c" exitCode=0 Nov 03 22:13:58.059163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:58.059126 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" Nov 03 22:13:58.059163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:58.059145 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" event={"ID":"d0dc65c0-bcea-4495-bf55-5095632b71b1","Type":"ContainerDied","Data":"afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c"} Nov 03 22:13:58.059412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:58.059185 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd57577d7-9mscc" event={"ID":"d0dc65c0-bcea-4495-bf55-5095632b71b1","Type":"ContainerDied","Data":"aa734418220c34c286aa79d5569883a058f7b47ddc706b869a6bd92d2128b652"} Nov 03 22:13:58.059412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:58.059204 4903 scope.go:122] "RemoveContainer" containerID="afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c" Nov 03 22:13:58.068241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:58.068226 4903 scope.go:122] "RemoveContainer" containerID="afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c" Nov 03 22:13:58.068584 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:13:58.068563 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c\": container with ID starting with afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c not found: ID does not exist" containerID="afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c" Nov 03 22:13:58.068635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:58.068592 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c"} err="failed to get container status \"afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c\": rpc error: code = NotFound desc = could not find container \"afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c\": container with ID starting with afea95ad1fbe65be06b65e32677f3db844ccbb0be793e9efdaf023174270256c not found: ID does not exist" Nov 03 22:13:58.114911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:58.114878 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fd57577d7-9mscc"] Nov 03 22:13:58.125278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:58.125253 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5fd57577d7-9mscc"] Nov 03 22:13:59.730156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:13:59.730111 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0dc65c0-bcea-4495-bf55-5095632b71b1" path="/var/lib/kubelet/pods/d0dc65c0-bcea-4495-bf55-5095632b71b1/volumes" Nov 03 22:14:18.380879 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:14:18.380840 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:14:29.170412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:14:29.170375 4903 generic.go:356] "Generic (PLEG): container finished" podID="33d39b02-7a8c-4edc-ae53-3ebd1e494f58" containerID="f99a44ce15ffde30f03e4d1f3f313f398c224127e352d5465a0ca122b827c935" exitCode=0 Nov 03 22:14:29.170888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:14:29.170451 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" event={"ID":"33d39b02-7a8c-4edc-ae53-3ebd1e494f58","Type":"ContainerDied","Data":"f99a44ce15ffde30f03e4d1f3f313f398c224127e352d5465a0ca122b827c935"} Nov 03 22:14:29.170888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:14:29.170768 4903 scope.go:122] "RemoveContainer" containerID="f99a44ce15ffde30f03e4d1f3f313f398c224127e352d5465a0ca122b827c935" Nov 03 22:14:30.175343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:14:30.175308 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r85cv" event={"ID":"33d39b02-7a8c-4edc-ae53-3ebd1e494f58","Type":"ContainerStarted","Data":"aa0dea202bf99c3fc52dcc92fef23393b07871f5063575034b00d353cfb3a96c"} Nov 03 22:14:48.381816 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:14:48.381778 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:15:18.381962 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:15:18.381884 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:15:48.383054 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:15:48.383012 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:16:18.382055 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:16:18.382013 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:16:21.314139 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.314105 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb"] Nov 03 22:16:21.314647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.314422 4903 cpu_manager.go:414] "containerMap: removing container" podUID="01e345d4-1cb8-4e4f-822d-d71b582b53b3" containerName="registry" Nov 03 22:16:21.314647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.314437 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e345d4-1cb8-4e4f-822d-d71b582b53b3" containerName="registry" Nov 03 22:16:21.314647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.314449 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d0dc65c0-bcea-4495-bf55-5095632b71b1" containerName="registry" Nov 03 22:16:21.314647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.314457 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0dc65c0-bcea-4495-bf55-5095632b71b1" containerName="registry" Nov 03 22:16:21.314647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.314534 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e345d4-1cb8-4e4f-822d-d71b582b53b3" containerName="registry" Nov 03 22:16:21.314647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.314544 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0dc65c0-bcea-4495-bf55-5095632b71b1" containerName="registry" Nov 03 22:16:21.317173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.317150 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.333145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.333118 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:16:21.333145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.333135 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:16:21.333287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.333167 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:16:21.333367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.333343 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:16:21.333401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.333390 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h854s\"" Nov 03 22:16:21.340681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.340655 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxfh5\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-kube-api-access-qxfh5\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.340767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.340693 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.340767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.340719 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.340835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.340785 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.340905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.340868 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.340942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.340902 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.340981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.340939 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.344910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.344888 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb"] Nov 03 22:16:21.441493 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.441458 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.441678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.441508 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.441678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.441630 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxfh5\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-kube-api-access-qxfh5\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.441678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.441672 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.441846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.441800 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.441903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.441841 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.441958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.441918 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.442504 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.442475 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.443864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.443836 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.444047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.444031 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.444163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.444143 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.444228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.444151 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.462695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.462669 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.462823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.462752 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxfh5\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-kube-api-access-qxfh5\") pod \"istiod-openshift-gateway-7876bfcb74-zqttb\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.627269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.627185 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:21.939418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:21.939388 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb"] Nov 03 22:16:21.943139 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:16:21.943111 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b6b815_fd6c_4b8a_959e_7686e043b0a4.slice/crio-45e14e0659b5d35700b4c2397b54ddfdfafe1dbc40b0224be204c64fb27444db WatchSource:0}: Error finding container 45e14e0659b5d35700b4c2397b54ddfdfafe1dbc40b0224be204c64fb27444db: Status 404 returned error can't find the container with id 45e14e0659b5d35700b4c2397b54ddfdfafe1dbc40b0224be204c64fb27444db Nov 03 22:16:22.569897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:22.569857 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" event={"ID":"e4b6b815-fd6c-4b8a-959e-7686e043b0a4","Type":"ContainerStarted","Data":"45e14e0659b5d35700b4c2397b54ddfdfafe1dbc40b0224be204c64fb27444db"} Nov 03 22:16:26.346265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:26.346231 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:16:26.346613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:26.346313 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:16:26.585527 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:26.585489 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" event={"ID":"e4b6b815-fd6c-4b8a-959e-7686e043b0a4","Type":"ContainerStarted","Data":"3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d"} Nov 03 22:16:26.585697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:26.585621 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:26.647826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:26.647718 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" podStartSLOduration=1.247245177 podStartE2EDuration="5.647702494s" totalImagesPullingTime="4.400457317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:16:26.636864014 +0000 UTC m=+213.040804988" watchObservedRunningTime="2026-11-03 22:16:26.647702494 +0000 UTC m=+213.051643469" Nov 03 22:16:27.590587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:27.590550 4903 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-zqttb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:16:27.590979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:27.590620 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" podUID="e4b6b815-fd6c-4b8a-959e-7686e043b0a4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:16:30.589973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:16:30.589943 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:16:48.382728 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:16:48.382689 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:17:09.970245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:09.970210 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749"] Nov 03 22:17:09.973318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:09.973283 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.001893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.001869 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 22:17:10.002026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.001895 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 22:17:10.006916 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.006891 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 22:17:10.007266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.007245 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 22:17:10.007850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.007830 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749"] Nov 03 22:17:10.098171 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.098134 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 22:17:10.098368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.098192 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wmvl7\"" Nov 03 22:17:10.135546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.135519 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.135641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.135550 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snn7f\" (UniqueName: \"kubernetes.io/projected/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-kube-api-access-snn7f\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.135641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.135569 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.135736 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.135666 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-metrics-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.135774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.135759 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-manifests\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.135830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.135817 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-tmp\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.236988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.236902 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.236988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.236933 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snn7f\" (UniqueName: \"kubernetes.io/projected/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-kube-api-access-snn7f\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.236988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.236952 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.236988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.236981 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-metrics-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.237339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.237097 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-manifests\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.237339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.237153 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-tmp\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.237575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.237534 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-manifests\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.239533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.239505 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-metrics-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.239622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.239514 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.239665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.239648 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-tmp\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.239803 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.239786 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-webhook-cert\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.258436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.258411 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snn7f\" (UniqueName: \"kubernetes.io/projected/ce5f3f14-641d-42af-b3c0-7ec51e27bd27-kube-api-access-snn7f\") pod \"opendatahub-operator-controller-manager-65cf5586b-d5749\" (UID: \"ce5f3f14-641d-42af-b3c0-7ec51e27bd27\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.287200 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.287172 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:10.642269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.642179 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749"] Nov 03 22:17:10.656985 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:17:10.656954 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5f3f14_641d_42af_b3c0_7ec51e27bd27.slice/crio-ee7e156740df7266d2e9fa1846b729af6d40d277e885e8e0fe7da2c6b2a7484d WatchSource:0}: Error finding container ee7e156740df7266d2e9fa1846b729af6d40d277e885e8e0fe7da2c6b2a7484d: Status 404 returned error can't find the container with id ee7e156740df7266d2e9fa1846b729af6d40d277e885e8e0fe7da2c6b2a7484d Nov 03 22:17:10.734567 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:10.734530 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" event={"ID":"ce5f3f14-641d-42af-b3c0-7ec51e27bd27","Type":"ContainerStarted","Data":"ee7e156740df7266d2e9fa1846b729af6d40d277e885e8e0fe7da2c6b2a7484d"} Nov 03 22:17:13.749308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:13.749255 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" event={"ID":"ce5f3f14-641d-42af-b3c0-7ec51e27bd27","Type":"ContainerStarted","Data":"b86eed10d618ca9c70da4b5b9554fd1a7e8e8fbc3b11524a3100b64feaa393d3"} Nov 03 22:17:14.754743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:14.754698 4903 generic.go:356] "Generic (PLEG): container finished" podID="ce5f3f14-641d-42af-b3c0-7ec51e27bd27" containerID="b86eed10d618ca9c70da4b5b9554fd1a7e8e8fbc3b11524a3100b64feaa393d3" exitCode=0 Nov 03 22:17:14.754743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:14.754786 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" event={"ID":"ce5f3f14-641d-42af-b3c0-7ec51e27bd27","Type":"ContainerDied","Data":"b86eed10d618ca9c70da4b5b9554fd1a7e8e8fbc3b11524a3100b64feaa393d3"} Nov 03 22:17:15.760575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:15.760542 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" event={"ID":"ce5f3f14-641d-42af-b3c0-7ec51e27bd27","Type":"ContainerStarted","Data":"b90ea050da88825060e512a0c4f930b3d8b32bd485c4bca82a325a6b3a9e0600"} Nov 03 22:17:15.760965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:15.760658 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:15.847502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:15.847445 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" podStartSLOduration=3.1373150069999998 podStartE2EDuration="6.84743207s" totalImagesPullingTime="3.231190977s" totalInitContainerRuntime="478.926086ms" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:15.836937874 +0000 UTC m=+262.240878849" watchObservedRunningTime="2026-11-03 22:17:15.84743207 +0000 UTC m=+262.251373045" Nov 03 22:17:18.382616 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:17:18.382570 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:17:26.767236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:26.767206 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65cf5586b-d5749" Nov 03 22:17:43.032900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.032863 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv"] Nov 03 22:17:43.036349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.036329 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.062609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.062573 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 22:17:43.062862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.062831 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qjnck\"" Nov 03 22:17:43.062970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.062886 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 22:17:43.063960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.063939 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv"] Nov 03 22:17:43.101868 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.101834 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5wz\" (UniqueName: \"kubernetes.io/projected/69f95d94-ca0e-475b-b197-956d4dc9556a-kube-api-access-kt5wz\") pod \"kube-auth-proxy-6b75ff7d64-dqsnv\" (UID: \"69f95d94-ca0e-475b-b197-956d4dc9556a\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.101868 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.101869 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69f95d94-ca0e-475b-b197-956d4dc9556a-tmp\") pod \"kube-auth-proxy-6b75ff7d64-dqsnv\" (UID: \"69f95d94-ca0e-475b-b197-956d4dc9556a\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.102056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.101888 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f95d94-ca0e-475b-b197-956d4dc9556a-tls-certs\") pod \"kube-auth-proxy-6b75ff7d64-dqsnv\" (UID: \"69f95d94-ca0e-475b-b197-956d4dc9556a\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.202465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.202425 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69f95d94-ca0e-475b-b197-956d4dc9556a-tmp\") pod \"kube-auth-proxy-6b75ff7d64-dqsnv\" (UID: \"69f95d94-ca0e-475b-b197-956d4dc9556a\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.202465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.202465 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f95d94-ca0e-475b-b197-956d4dc9556a-tls-certs\") pod \"kube-auth-proxy-6b75ff7d64-dqsnv\" (UID: \"69f95d94-ca0e-475b-b197-956d4dc9556a\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.202684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.202536 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5wz\" (UniqueName: \"kubernetes.io/projected/69f95d94-ca0e-475b-b197-956d4dc9556a-kube-api-access-kt5wz\") pod \"kube-auth-proxy-6b75ff7d64-dqsnv\" (UID: \"69f95d94-ca0e-475b-b197-956d4dc9556a\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.204852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.204819 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69f95d94-ca0e-475b-b197-956d4dc9556a-tmp\") pod \"kube-auth-proxy-6b75ff7d64-dqsnv\" (UID: \"69f95d94-ca0e-475b-b197-956d4dc9556a\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.205086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.205068 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f95d94-ca0e-475b-b197-956d4dc9556a-tls-certs\") pod \"kube-auth-proxy-6b75ff7d64-dqsnv\" (UID: \"69f95d94-ca0e-475b-b197-956d4dc9556a\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.227542 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.227514 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5wz\" (UniqueName: \"kubernetes.io/projected/69f95d94-ca0e-475b-b197-956d4dc9556a-kube-api-access-kt5wz\") pod \"kube-auth-proxy-6b75ff7d64-dqsnv\" (UID: \"69f95d94-ca0e-475b-b197-956d4dc9556a\") " pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.348435 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.348363 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" Nov 03 22:17:43.645820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.645681 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv"] Nov 03 22:17:43.648374 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:17:43.648347 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f95d94_ca0e_475b_b197_956d4dc9556a.slice/crio-e60e37c220276e23c8f16566e8aeb2545d84b906cbd27048d52d9cce001b328d WatchSource:0}: Error finding container e60e37c220276e23c8f16566e8aeb2545d84b906cbd27048d52d9cce001b328d: Status 404 returned error can't find the container with id e60e37c220276e23c8f16566e8aeb2545d84b906cbd27048d52d9cce001b328d Nov 03 22:17:43.866092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:43.866059 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" event={"ID":"69f95d94-ca0e-475b-b197-956d4dc9556a","Type":"ContainerStarted","Data":"e60e37c220276e23c8f16566e8aeb2545d84b906cbd27048d52d9cce001b328d"} Nov 03 22:17:46.880183 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:46.880147 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" event={"ID":"69f95d94-ca0e-475b-b197-956d4dc9556a","Type":"ContainerStarted","Data":"7553de1e4a5c8d6bc49cfe53d59cefb39e03fee10dcd67fe06dfd3de80c9ad50"} Nov 03 22:17:46.931328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:46.931254 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b75ff7d64-dqsnv" podStartSLOduration=1.596268713 podStartE2EDuration="3.931238594s" totalImagesPullingTime="2.334969881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:17:46.930620805 +0000 UTC m=+293.334561780" watchObservedRunningTime="2026-11-03 22:17:46.931238594 +0000 UTC m=+293.335179560" Nov 03 22:17:48.383275 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:17:48.383242 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:17:53.750538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:53.750512 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:17:53.751192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:53.751172 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:17:53.757635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:53.757615 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:17:53.757886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:17:53.757870 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:18:11.839592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.839519 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zkzdg"] Nov 03 22:18:11.843330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.843286 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:18:11.843494 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.843305 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" Nov 03 22:18:11.846502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.846482 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:11.860825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.860797 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zkzdg"] Nov 03 22:18:11.870470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.870443 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9zcrl\"" Nov 03 22:18:11.870597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.870471 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 22:18:11.870597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.870504 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zzppl\"" Nov 03 22:18:11.870597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.870521 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 22:18:11.870597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.870529 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 22:18:11.871554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.871534 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:18:11.937981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.937947 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:18:11.950083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.950054 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2lpf\" (UniqueName: \"kubernetes.io/projected/d25b1701-3aca-4d22-a013-0fddfc92ab62-kube-api-access-f2lpf\") pod \"authorino-56c6c9ddd5-zkzdg\" (UID: \"d25b1701-3aca-4d22-a013-0fddfc92ab62\") " pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" Nov 03 22:18:11.950162 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.950125 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8sw9\" (UniqueName: \"kubernetes.io/projected/7a093438-7423-4f79-903b-b88a4e05b08c-kube-api-access-q8sw9\") pod \"limitador-limitador-7c9b87d574-dg6s6\" (UID: \"7a093438-7423-4f79-903b-b88a4e05b08c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:11.950233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:11.950215 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a093438-7423-4f79-903b-b88a4e05b08c-config-file\") pod \"limitador-limitador-7c9b87d574-dg6s6\" (UID: \"7a093438-7423-4f79-903b-b88a4e05b08c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:12.051618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.051580 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a093438-7423-4f79-903b-b88a4e05b08c-config-file\") pod \"limitador-limitador-7c9b87d574-dg6s6\" (UID: \"7a093438-7423-4f79-903b-b88a4e05b08c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:12.051816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.051631 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2lpf\" (UniqueName: \"kubernetes.io/projected/d25b1701-3aca-4d22-a013-0fddfc92ab62-kube-api-access-f2lpf\") pod \"authorino-56c6c9ddd5-zkzdg\" (UID: \"d25b1701-3aca-4d22-a013-0fddfc92ab62\") " pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" Nov 03 22:18:12.051816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.051714 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8sw9\" (UniqueName: \"kubernetes.io/projected/7a093438-7423-4f79-903b-b88a4e05b08c-kube-api-access-q8sw9\") pod \"limitador-limitador-7c9b87d574-dg6s6\" (UID: \"7a093438-7423-4f79-903b-b88a4e05b08c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:12.052246 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.052225 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7a093438-7423-4f79-903b-b88a4e05b08c-config-file\") pod \"limitador-limitador-7c9b87d574-dg6s6\" (UID: \"7a093438-7423-4f79-903b-b88a4e05b08c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:12.073410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.073380 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2lpf\" (UniqueName: \"kubernetes.io/projected/d25b1701-3aca-4d22-a013-0fddfc92ab62-kube-api-access-f2lpf\") pod \"authorino-56c6c9ddd5-zkzdg\" (UID: \"d25b1701-3aca-4d22-a013-0fddfc92ab62\") " pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" Nov 03 22:18:12.073527 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.073460 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8sw9\" (UniqueName: \"kubernetes.io/projected/7a093438-7423-4f79-903b-b88a4e05b08c-kube-api-access-q8sw9\") pod \"limitador-limitador-7c9b87d574-dg6s6\" (UID: \"7a093438-7423-4f79-903b-b88a4e05b08c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:12.165300 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.165265 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" Nov 03 22:18:12.174335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.174310 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:12.461391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.461223 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zkzdg"] Nov 03 22:18:12.464258 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:18:12.464233 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd25b1701_3aca_4d22_a013_0fddfc92ab62.slice/crio-4e90ccaa687f075ee08b7087bc3f799d9215d111b460b63d80edf66e9497151f WatchSource:0}: Error finding container 4e90ccaa687f075ee08b7087bc3f799d9215d111b460b63d80edf66e9497151f: Status 404 returned error can't find the container with id 4e90ccaa687f075ee08b7087bc3f799d9215d111b460b63d80edf66e9497151f Nov 03 22:18:12.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.465575 4903 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:18:12.474457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.474435 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:18:12.477403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:18:12.477382 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a093438_7423_4f79_903b_b88a4e05b08c.slice/crio-f6dbeec1a6c9ceea42d7729e7c2497f7ac84ade503421f1782be11c5f8ccd859 WatchSource:0}: Error finding container f6dbeec1a6c9ceea42d7729e7c2497f7ac84ade503421f1782be11c5f8ccd859: Status 404 returned error can't find the container with id f6dbeec1a6c9ceea42d7729e7c2497f7ac84ade503421f1782be11c5f8ccd859 Nov 03 22:18:12.982061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.982013 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" event={"ID":"7a093438-7423-4f79-903b-b88a4e05b08c","Type":"ContainerStarted","Data":"f6dbeec1a6c9ceea42d7729e7c2497f7ac84ade503421f1782be11c5f8ccd859"} Nov 03 22:18:12.983337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:12.983273 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" event={"ID":"d25b1701-3aca-4d22-a013-0fddfc92ab62","Type":"ContainerStarted","Data":"4e90ccaa687f075ee08b7087bc3f799d9215d111b460b63d80edf66e9497151f"} Nov 03 22:18:15.999061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:15.999019 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" event={"ID":"7a093438-7423-4f79-903b-b88a4e05b08c","Type":"ContainerStarted","Data":"4134db8f45c9593114863256e877198461c77cfa62324fff744b8ed226d4ffd2"} Nov 03 22:18:15.999615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:15.999181 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:16.000561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:16.000538 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" event={"ID":"d25b1701-3aca-4d22-a013-0fddfc92ab62","Type":"ContainerStarted","Data":"d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521"} Nov 03 22:18:16.036225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:16.036182 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" podStartSLOduration=2.572530817 podStartE2EDuration="5.036169085s" totalImagesPullingTime="2.463638268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:16.035415352 +0000 UTC m=+322.439356328" watchObservedRunningTime="2026-11-03 22:18:16.036169085 +0000 UTC m=+322.440110054" Nov 03 22:18:16.069895 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:16.069841 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" podStartSLOduration=2.536469571 podStartE2EDuration="5.069825587s" totalImagesPullingTime="2.533356016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:16.069365583 +0000 UTC m=+322.473306558" watchObservedRunningTime="2026-11-03 22:18:16.069825587 +0000 UTC m=+322.473766563" Nov 03 22:18:18.383108 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:18:18.383067 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:18:27.005851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:27.005816 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-dg6s6" Nov 03 22:18:48.382786 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:18:48.382748 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:18:48.858077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:48.858044 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb"] Nov 03 22:18:48.858412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:48.858251 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" podUID="e4b6b815-fd6c-4b8a-959e-7686e043b0a4" containerName="discovery" containerID="cri-o://3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d" gracePeriod=30 Nov 03 22:18:49.103882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.103850 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:18:49.135973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.135891 4903 generic.go:356] "Generic (PLEG): container finished" podID="e4b6b815-fd6c-4b8a-959e-7686e043b0a4" containerID="3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d" exitCode=0 Nov 03 22:18:49.135973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.135935 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" Nov 03 22:18:49.136141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.135971 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" event={"ID":"e4b6b815-fd6c-4b8a-959e-7686e043b0a4","Type":"ContainerDied","Data":"3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d"} Nov 03 22:18:49.136141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.136049 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb" event={"ID":"e4b6b815-fd6c-4b8a-959e-7686e043b0a4","Type":"ContainerDied","Data":"45e14e0659b5d35700b4c2397b54ddfdfafe1dbc40b0224be204c64fb27444db"} Nov 03 22:18:49.136141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.136071 4903 scope.go:122] "RemoveContainer" containerID="3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d" Nov 03 22:18:49.143777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.143759 4903 scope.go:122] "RemoveContainer" containerID="3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d" Nov 03 22:18:49.144055 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:18:49.144025 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d\": container with ID starting with 3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d not found: ID does not exist" containerID="3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d" Nov 03 22:18:49.144153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.144060 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d"} err="failed to get container status \"3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d\": rpc error: code = NotFound desc = could not find container \"3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d\": container with ID starting with 3b3a686c458f9c23d3e5293aa0b7e9be886f88a235eda826da17e683e395fc9d not found: ID does not exist" Nov 03 22:18:49.156744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.156722 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-ca-configmap\") pod \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " Nov 03 22:18:49.156870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.156784 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-kubeconfig\") pod \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " Nov 03 22:18:49.156870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.156803 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-dns-cert\") pod \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " Nov 03 22:18:49.156870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.156839 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-cacerts\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-cacerts\") pod \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " Nov 03 22:18:49.157026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.156882 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-local-certs\") pod \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " Nov 03 22:18:49.157026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.156918 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-token\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-token\") pod \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " Nov 03 22:18:49.157026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.156961 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-kube-api-access-qxfh5\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-kube-api-access-qxfh5\") pod \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\" (UID: \"e4b6b815-fd6c-4b8a-959e-7686e043b0a4\") " Nov 03 22:18:49.157182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.157158 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-ca-configmap" pod "e4b6b815-fd6c-4b8a-959e-7686e043b0a4" (UID: "e4b6b815-fd6c-4b8a-959e-7686e043b0a4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:18:49.159416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.159389 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-dns-cert" pod "e4b6b815-fd6c-4b8a-959e-7686e043b0a4" (UID: "e4b6b815-fd6c-4b8a-959e-7686e043b0a4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:18:49.159514 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.159415 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-kubeconfig" pod "e4b6b815-fd6c-4b8a-959e-7686e043b0a4" (UID: "e4b6b815-fd6c-4b8a-959e-7686e043b0a4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:18:49.159651 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.159632 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-cacerts" pod "e4b6b815-fd6c-4b8a-959e-7686e043b0a4" (UID: "e4b6b815-fd6c-4b8a-959e-7686e043b0a4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:18:49.159835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.159817 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-local-certs" pod "e4b6b815-fd6c-4b8a-959e-7686e043b0a4" (UID: "e4b6b815-fd6c-4b8a-959e-7686e043b0a4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:18:49.160443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.160419 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-token" pod "e4b6b815-fd6c-4b8a-959e-7686e043b0a4" (UID: "e4b6b815-fd6c-4b8a-959e-7686e043b0a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:18:49.160641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.160626 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-kube-api-access-qxfh5" pod "e4b6b815-fd6c-4b8a-959e-7686e043b0a4" (UID: "e4b6b815-fd6c-4b8a-959e-7686e043b0a4"). InnerVolumeSpecName "kube-api-access-qxfh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:18:49.258190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.258147 4903 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-local-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:18:49.258190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.258181 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:18:49.258190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.258196 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxfh5\" (UniqueName: \"kubernetes.io/projected/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-kube-api-access-qxfh5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:18:49.258461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.258209 4903 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-ca-configmap\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:18:49.258461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.258223 4903 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-kubeconfig\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:18:49.258461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.258236 4903 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-istio-csr-dns-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:18:49.258461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.258246 4903 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4b6b815-fd6c-4b8a-959e-7686e043b0a4-cacerts\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:18:49.475710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.475669 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb"] Nov 03 22:18:49.480157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.480129 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-zqttb"] Nov 03 22:18:49.731341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:49.731256 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b6b815-fd6c-4b8a-959e-7686e043b0a4" path="/var/lib/kubelet/pods/e4b6b815-fd6c-4b8a-959e-7686e043b0a4/volumes" Nov 03 22:18:50.803980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.803939 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rrnwb"] Nov 03 22:18:50.804492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.804410 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e4b6b815-fd6c-4b8a-959e-7686e043b0a4" containerName="discovery" Nov 03 22:18:50.804492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.804429 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b6b815-fd6c-4b8a-959e-7686e043b0a4" containerName="discovery" Nov 03 22:18:50.804605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.804541 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b6b815-fd6c-4b8a-959e-7686e043b0a4" containerName="discovery" Nov 03 22:18:50.807990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.807970 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:50.828487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.828457 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rrnwb"] Nov 03 22:18:50.857737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.857701 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 22:18:50.858204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.858190 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 22:18:50.858312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.858243 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rnrkl\"" Nov 03 22:18:50.867227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.867191 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 22:18:50.867325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.867214 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 22:18:50.867325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.867230 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:18:50.869629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.869613 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxdsm\" (UniqueName: \"kubernetes.io/projected/4030adbd-d677-4ff1-8c77-6617d87a1c9e-kube-api-access-gxdsm\") pod \"maas-controller-579cf7c6d-rrnwb\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:50.869673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.869665 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-webhook-cert\") pod \"maas-controller-579cf7c6d-rrnwb\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:50.869712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.869693 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-metrics-certs\") pod \"maas-controller-579cf7c6d-rrnwb\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:50.970119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.970082 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxdsm\" (UniqueName: \"kubernetes.io/projected/4030adbd-d677-4ff1-8c77-6617d87a1c9e-kube-api-access-gxdsm\") pod \"maas-controller-579cf7c6d-rrnwb\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:50.970320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.970142 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-webhook-cert\") pod \"maas-controller-579cf7c6d-rrnwb\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:50.970320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.970171 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-metrics-certs\") pod \"maas-controller-579cf7c6d-rrnwb\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:50.972643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.972616 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-metrics-certs\") pod \"maas-controller-579cf7c6d-rrnwb\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:50.972769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.972695 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-webhook-cert\") pod \"maas-controller-579cf7c6d-rrnwb\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:50.991544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:50.991519 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxdsm\" (UniqueName: \"kubernetes.io/projected/4030adbd-d677-4ff1-8c77-6617d87a1c9e-kube-api-access-gxdsm\") pod \"maas-controller-579cf7c6d-rrnwb\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:51.119640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.119557 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:51.156551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.156517 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d9589f759-8srgp"] Nov 03 22:18:51.159412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.159384 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.185418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.183949 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9589f759-8srgp"] Nov 03 22:18:51.274829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.274793 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2km\" (UniqueName: \"kubernetes.io/projected/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-kube-api-access-8t2km\") pod \"maas-controller-6d9589f759-8srgp\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.275003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.274846 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-metrics-certs\") pod \"maas-controller-6d9589f759-8srgp\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.275003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.274939 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-webhook-cert\") pod \"maas-controller-6d9589f759-8srgp\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.376010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.375943 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2km\" (UniqueName: \"kubernetes.io/projected/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-kube-api-access-8t2km\") pod \"maas-controller-6d9589f759-8srgp\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.376010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.375988 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-metrics-certs\") pod \"maas-controller-6d9589f759-8srgp\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.376187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.376016 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-webhook-cert\") pod \"maas-controller-6d9589f759-8srgp\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.378456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.378429 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-metrics-certs\") pod \"maas-controller-6d9589f759-8srgp\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.378564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.378434 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-webhook-cert\") pod \"maas-controller-6d9589f759-8srgp\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.397573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.397545 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2km\" (UniqueName: \"kubernetes.io/projected/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-kube-api-access-8t2km\") pod \"maas-controller-6d9589f759-8srgp\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.414510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.414491 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rrnwb"] Nov 03 22:18:51.417171 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:18:51.417145 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4030adbd_d677_4ff1_8c77_6617d87a1c9e.slice/crio-163d2ad036d4fcd39929f11dfb658e57b352b39a01412c2e4b88849995ce2e3d WatchSource:0}: Error finding container 163d2ad036d4fcd39929f11dfb658e57b352b39a01412c2e4b88849995ce2e3d: Status 404 returned error can't find the container with id 163d2ad036d4fcd39929f11dfb658e57b352b39a01412c2e4b88849995ce2e3d Nov 03 22:18:51.476607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.476572 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:51.769598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:51.769574 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d9589f759-8srgp"] Nov 03 22:18:51.772191 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:18:51.772162 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40c3a4f_13bb_4d9a_a3b6_9ae260278d88.slice/crio-ee49703ba68be15a51603032e41cf942f5e93497ebff8bd85b521b36c25c1d1c WatchSource:0}: Error finding container ee49703ba68be15a51603032e41cf942f5e93497ebff8bd85b521b36c25c1d1c: Status 404 returned error can't find the container with id ee49703ba68be15a51603032e41cf942f5e93497ebff8bd85b521b36c25c1d1c Nov 03 22:18:52.154253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:52.154215 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" event={"ID":"4030adbd-d677-4ff1-8c77-6617d87a1c9e","Type":"ContainerStarted","Data":"163d2ad036d4fcd39929f11dfb658e57b352b39a01412c2e4b88849995ce2e3d"} Nov 03 22:18:52.156184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:52.156155 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-8srgp" event={"ID":"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88","Type":"ContainerStarted","Data":"ee49703ba68be15a51603032e41cf942f5e93497ebff8bd85b521b36c25c1d1c"} Nov 03 22:18:54.190220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:54.190194 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 22:18:55.171179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.171146 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" event={"ID":"4030adbd-d677-4ff1-8c77-6617d87a1c9e","Type":"ContainerStarted","Data":"5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7"} Nov 03 22:18:55.171417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.171269 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:18:55.172949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.172923 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-8srgp" event={"ID":"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88","Type":"ContainerStarted","Data":"029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79"} Nov 03 22:18:55.173071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.173022 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:18:55.210273 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.210228 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" podStartSLOduration=2.45062517 podStartE2EDuration="5.210214346s" totalImagesPullingTime="2.759589176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:55.209476716 +0000 UTC m=+361.613417690" watchObservedRunningTime="2026-11-03 22:18:55.210214346 +0000 UTC m=+361.614155318" Nov 03 22:18:55.248591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.248537 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6d9589f759-8srgp" podStartSLOduration=1.679608671 podStartE2EDuration="4.24852718s" totalImagesPullingTime="2.568918509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:55.247971168 +0000 UTC m=+361.651912144" watchObservedRunningTime="2026-11-03 22:18:55.24852718 +0000 UTC m=+361.652468155" Nov 03 22:18:55.901237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.901191 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576"] Nov 03 22:18:55.903755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.903736 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:55.922159 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.922139 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 22:18:55.932473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.932451 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 22:18:55.932600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.932482 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 22:18:55.932600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.932511 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 22:18:55.933244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.933227 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-28746\"" Nov 03 22:18:55.933281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.933263 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 22:18:55.933730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:55.933714 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576"] Nov 03 22:18:56.015623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.015590 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcznf\" (UniqueName: \"kubernetes.io/projected/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-kube-api-access-kcznf\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.015792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.015635 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-cert\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.015792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.015725 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-metrics-certs\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.015792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.015777 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-manager-config\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.116659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.116622 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-cert\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.116826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.116688 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-metrics-certs\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.116826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.116729 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-manager-config\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.116826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.116760 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcznf\" (UniqueName: \"kubernetes.io/projected/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-kube-api-access-kcznf\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.117444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.117425 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-manager-config\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.119160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.119135 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-metrics-certs\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.119257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.119142 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-cert\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.148617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.148592 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcznf\" (UniqueName: \"kubernetes.io/projected/fb747399-6b70-42b4-8309-5a2ceb2fe0ec-kube-api-access-kcznf\") pod \"jobset-controller-manager-6449b4c8fb-kt576\" (UID: \"fb747399-6b70-42b4-8309-5a2ceb2fe0ec\") " pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.214845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.214772 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:56.506088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:56.505993 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576"] Nov 03 22:18:56.508216 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:18:56.508189 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb747399_6b70_42b4_8309_5a2ceb2fe0ec.slice/crio-07a8a5a59867e5328a58905235145bfa7c05b22bfaa3c9978e64ccbf45e5764d WatchSource:0}: Error finding container 07a8a5a59867e5328a58905235145bfa7c05b22bfaa3c9978e64ccbf45e5764d: Status 404 returned error can't find the container with id 07a8a5a59867e5328a58905235145bfa7c05b22bfaa3c9978e64ccbf45e5764d Nov 03 22:18:57.182076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:57.182031 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" event={"ID":"fb747399-6b70-42b4-8309-5a2ceb2fe0ec","Type":"ContainerStarted","Data":"07a8a5a59867e5328a58905235145bfa7c05b22bfaa3c9978e64ccbf45e5764d"} Nov 03 22:18:59.191957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:59.191857 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" event={"ID":"fb747399-6b70-42b4-8309-5a2ceb2fe0ec","Type":"ContainerStarted","Data":"04559fe7585e613c83eeb7789e6daddf9a4f69c69607e4fd6f0dd403206f017f"} Nov 03 22:18:59.191957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:59.191929 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:18:59.229384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:18:59.229325 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" podStartSLOduration=1.801877396 podStartE2EDuration="4.229285595s" totalImagesPullingTime="2.427408199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:18:59.228369597 +0000 UTC m=+365.632310572" watchObservedRunningTime="2026-11-03 22:18:59.229285595 +0000 UTC m=+365.633226571" Nov 03 22:19:06.179519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.179487 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:19:06.180037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.180012 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 22:19:06.291146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.291111 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rrnwb"] Nov 03 22:19:06.291389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.291363 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" podUID="4030adbd-d677-4ff1-8c77-6617d87a1c9e" containerName="manager" containerID="cri-o://5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7" gracePeriod=30 Nov 03 22:19:06.526262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.526234 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:19:06.608890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.608853 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4030adbd-d677-4ff1-8c77-6617d87a1c9e-kube-api-access-gxdsm\" (UniqueName: \"kubernetes.io/projected/4030adbd-d677-4ff1-8c77-6617d87a1c9e-kube-api-access-gxdsm\") pod \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " Nov 03 22:19:06.609059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.608900 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-webhook-cert\") pod \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " Nov 03 22:19:06.609059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.608948 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-metrics-certs\") pod \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\" (UID: \"4030adbd-d677-4ff1-8c77-6617d87a1c9e\") " Nov 03 22:19:06.611182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.611134 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-webhook-cert" pod "4030adbd-d677-4ff1-8c77-6617d87a1c9e" (UID: "4030adbd-d677-4ff1-8c77-6617d87a1c9e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:19:06.611182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.611169 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4030adbd-d677-4ff1-8c77-6617d87a1c9e-kube-api-access-gxdsm" pod "4030adbd-d677-4ff1-8c77-6617d87a1c9e" (UID: "4030adbd-d677-4ff1-8c77-6617d87a1c9e"). InnerVolumeSpecName "kube-api-access-gxdsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:06.611624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.611600 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-metrics-certs" pod "4030adbd-d677-4ff1-8c77-6617d87a1c9e" (UID: "4030adbd-d677-4ff1-8c77-6617d87a1c9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:19:06.710037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.709954 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxdsm\" (UniqueName: \"kubernetes.io/projected/4030adbd-d677-4ff1-8c77-6617d87a1c9e-kube-api-access-gxdsm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:19:06.710037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.709987 4903 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-webhook-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:19:06.710037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:06.709997 4903 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4030adbd-d677-4ff1-8c77-6617d87a1c9e-metrics-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:19:07.226725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.226691 4903 generic.go:356] "Generic (PLEG): container finished" podID="4030adbd-d677-4ff1-8c77-6617d87a1c9e" containerID="5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7" exitCode=0 Nov 03 22:19:07.227157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.226740 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" Nov 03 22:19:07.227157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.226777 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" event={"ID":"4030adbd-d677-4ff1-8c77-6617d87a1c9e","Type":"ContainerDied","Data":"5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7"} Nov 03 22:19:07.227157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.226829 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rrnwb" event={"ID":"4030adbd-d677-4ff1-8c77-6617d87a1c9e","Type":"ContainerDied","Data":"163d2ad036d4fcd39929f11dfb658e57b352b39a01412c2e4b88849995ce2e3d"} Nov 03 22:19:07.227157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.226853 4903 scope.go:122] "RemoveContainer" containerID="5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7" Nov 03 22:19:07.235575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.235560 4903 scope.go:122] "RemoveContainer" containerID="5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7" Nov 03 22:19:07.235844 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:19:07.235827 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7\": container with ID starting with 5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7 not found: ID does not exist" containerID="5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7" Nov 03 22:19:07.235890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.235855 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7"} err="failed to get container status \"5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7\": rpc error: code = NotFound desc = could not find container \"5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7\": container with ID starting with 5115b9e98d55f59c9881e15a90952d00f30e813b3d3d042eb2f7ec28319f76f7 not found: ID does not exist" Nov 03 22:19:07.271451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.271425 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rrnwb"] Nov 03 22:19:07.281703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.281679 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rrnwb"] Nov 03 22:19:07.730733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:07.730701 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4030adbd-d677-4ff1-8c77-6617d87a1c9e" path="/var/lib/kubelet/pods/4030adbd-d677-4ff1-8c77-6617d87a1c9e/volumes" Nov 03 22:19:10.201633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:10.201605 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6449b4c8fb-kt576" Nov 03 22:19:18.382441 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:19:18.382403 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:19:27.613665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:27.613625 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:19:35.562473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.562439 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8l525"] Nov 03 22:19:35.562856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.562789 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4030adbd-d677-4ff1-8c77-6617d87a1c9e" containerName="manager" Nov 03 22:19:35.562856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.562803 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4030adbd-d677-4ff1-8c77-6617d87a1c9e" containerName="manager" Nov 03 22:19:35.562926 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.562867 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4030adbd-d677-4ff1-8c77-6617d87a1c9e" containerName="manager" Nov 03 22:19:35.569218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.569199 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-8l525" Nov 03 22:19:35.585530 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.585507 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8l525"] Nov 03 22:19:35.637640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.637604 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlm2q\" (UniqueName: \"kubernetes.io/projected/8347eda6-dc7f-426b-9254-a395f1fa86fa-kube-api-access-tlm2q\") pod \"authorino-6f6944bcdd-8l525\" (UID: \"8347eda6-dc7f-426b-9254-a395f1fa86fa\") " pod="kuadrant-system/authorino-6f6944bcdd-8l525" Nov 03 22:19:35.738276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.738245 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlm2q\" (UniqueName: \"kubernetes.io/projected/8347eda6-dc7f-426b-9254-a395f1fa86fa-kube-api-access-tlm2q\") pod \"authorino-6f6944bcdd-8l525\" (UID: \"8347eda6-dc7f-426b-9254-a395f1fa86fa\") " pod="kuadrant-system/authorino-6f6944bcdd-8l525" Nov 03 22:19:35.750229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.750206 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8l525"] Nov 03 22:19:35.750402 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:19:35.750383 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-tlm2q], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-8l525" podUID="8347eda6-dc7f-426b-9254-a395f1fa86fa" Nov 03 22:19:35.764980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.764955 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlm2q\" (UniqueName: \"kubernetes.io/projected/8347eda6-dc7f-426b-9254-a395f1fa86fa-kube-api-access-tlm2q\") pod \"authorino-6f6944bcdd-8l525\" (UID: \"8347eda6-dc7f-426b-9254-a395f1fa86fa\") " pod="kuadrant-system/authorino-6f6944bcdd-8l525" Nov 03 22:19:35.783464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.783431 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fjzr6"] Nov 03 22:19:35.785888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.785872 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:35.799044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.799025 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 22:19:35.804115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.804091 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fjzr6"] Nov 03 22:19:35.839474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.839392 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/95ab0df6-fcba-433b-8e25-7298d6c6cc14-tls-cert\") pod \"authorino-6889cbd8c4-fjzr6\" (UID: \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\") " pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:35.839474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.839464 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrwmn\" (UniqueName: \"kubernetes.io/projected/95ab0df6-fcba-433b-8e25-7298d6c6cc14-kube-api-access-nrwmn\") pod \"authorino-6889cbd8c4-fjzr6\" (UID: \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\") " pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:35.926057 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.926025 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fjzr6"] Nov 03 22:19:35.926220 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:19:35.926198 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nrwmn tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" podUID="95ab0df6-fcba-433b-8e25-7298d6c6cc14" Nov 03 22:19:35.940355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.940331 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/95ab0df6-fcba-433b-8e25-7298d6c6cc14-tls-cert\") pod \"authorino-6889cbd8c4-fjzr6\" (UID: \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\") " pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:35.940487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.940383 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrwmn\" (UniqueName: \"kubernetes.io/projected/95ab0df6-fcba-433b-8e25-7298d6c6cc14-kube-api-access-nrwmn\") pod \"authorino-6889cbd8c4-fjzr6\" (UID: \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\") " pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:35.942877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.942856 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/95ab0df6-fcba-433b-8e25-7298d6c6cc14-tls-cert\") pod \"authorino-6889cbd8c4-fjzr6\" (UID: \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\") " pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:35.953634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.953607 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bf5dfd866-kkskx"] Nov 03 22:19:35.956033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.956015 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:19:35.964023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.964004 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrwmn\" (UniqueName: \"kubernetes.io/projected/95ab0df6-fcba-433b-8e25-7298d6c6cc14-kube-api-access-nrwmn\") pod \"authorino-6889cbd8c4-fjzr6\" (UID: \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\") " pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:35.973963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:35.973938 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bf5dfd866-kkskx"] Nov 03 22:19:36.041355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.041255 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95f2n\" (UniqueName: \"kubernetes.io/projected/5762840f-90fb-4277-a566-47c3438299d7-kube-api-access-95f2n\") pod \"authorino-6bf5dfd866-kkskx\" (UID: \"5762840f-90fb-4277-a566-47c3438299d7\") " pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:19:36.041355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.041335 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5762840f-90fb-4277-a566-47c3438299d7-tls-cert\") pod \"authorino-6bf5dfd866-kkskx\" (UID: \"5762840f-90fb-4277-a566-47c3438299d7\") " pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:19:36.142813 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.142731 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95f2n\" (UniqueName: \"kubernetes.io/projected/5762840f-90fb-4277-a566-47c3438299d7-kube-api-access-95f2n\") pod \"authorino-6bf5dfd866-kkskx\" (UID: \"5762840f-90fb-4277-a566-47c3438299d7\") " pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:19:36.142813 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.142789 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5762840f-90fb-4277-a566-47c3438299d7-tls-cert\") pod \"authorino-6bf5dfd866-kkskx\" (UID: \"5762840f-90fb-4277-a566-47c3438299d7\") " pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:19:36.145146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.145120 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5762840f-90fb-4277-a566-47c3438299d7-tls-cert\") pod \"authorino-6bf5dfd866-kkskx\" (UID: \"5762840f-90fb-4277-a566-47c3438299d7\") " pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:19:36.163813 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.163786 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95f2n\" (UniqueName: \"kubernetes.io/projected/5762840f-90fb-4277-a566-47c3438299d7-kube-api-access-95f2n\") pod \"authorino-6bf5dfd866-kkskx\" (UID: \"5762840f-90fb-4277-a566-47c3438299d7\") " pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:19:36.266599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.266560 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:19:36.342101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.342071 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-8l525" Nov 03 22:19:36.342261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.342071 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:36.349976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.349954 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-8l525" Nov 03 22:19:36.352870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.352852 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:36.445383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.445353 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8347eda6-dc7f-426b-9254-a395f1fa86fa-kube-api-access-tlm2q\" (UniqueName: \"kubernetes.io/projected/8347eda6-dc7f-426b-9254-a395f1fa86fa-kube-api-access-tlm2q\") pod \"8347eda6-dc7f-426b-9254-a395f1fa86fa\" (UID: \"8347eda6-dc7f-426b-9254-a395f1fa86fa\") " Nov 03 22:19:36.445567 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.445430 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ab0df6-fcba-433b-8e25-7298d6c6cc14-tls-cert\" (UniqueName: \"kubernetes.io/secret/95ab0df6-fcba-433b-8e25-7298d6c6cc14-tls-cert\") pod \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\" (UID: \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\") " Nov 03 22:19:36.445567 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.445489 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ab0df6-fcba-433b-8e25-7298d6c6cc14-kube-api-access-nrwmn\" (UniqueName: \"kubernetes.io/projected/95ab0df6-fcba-433b-8e25-7298d6c6cc14-kube-api-access-nrwmn\") pod \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\" (UID: \"95ab0df6-fcba-433b-8e25-7298d6c6cc14\") " Nov 03 22:19:36.447686 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.447652 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ab0df6-fcba-433b-8e25-7298d6c6cc14-tls-cert" pod "95ab0df6-fcba-433b-8e25-7298d6c6cc14" (UID: "95ab0df6-fcba-433b-8e25-7298d6c6cc14"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:19:36.447804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.447711 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8347eda6-dc7f-426b-9254-a395f1fa86fa-kube-api-access-tlm2q" pod "8347eda6-dc7f-426b-9254-a395f1fa86fa" (UID: "8347eda6-dc7f-426b-9254-a395f1fa86fa"). InnerVolumeSpecName "kube-api-access-tlm2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:36.447804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.447742 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ab0df6-fcba-433b-8e25-7298d6c6cc14-kube-api-access-nrwmn" pod "95ab0df6-fcba-433b-8e25-7298d6c6cc14" (UID: "95ab0df6-fcba-433b-8e25-7298d6c6cc14"). InnerVolumeSpecName "kube-api-access-nrwmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:36.546145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.546110 4903 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/95ab0df6-fcba-433b-8e25-7298d6c6cc14-tls-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:19:36.546145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.546138 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrwmn\" (UniqueName: \"kubernetes.io/projected/95ab0df6-fcba-433b-8e25-7298d6c6cc14-kube-api-access-nrwmn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:19:36.546145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.546149 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlm2q\" (UniqueName: \"kubernetes.io/projected/8347eda6-dc7f-426b-9254-a395f1fa86fa-kube-api-access-tlm2q\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:19:36.556476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:36.556444 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bf5dfd866-kkskx"] Nov 03 22:19:36.559421 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:19:36.559394 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5762840f_90fb_4277_a566_47c3438299d7.slice/crio-f20cfe17e11643433b2160fcf9ef7c4772414bc0a986f56319bdbbe4f097d0b2 WatchSource:0}: Error finding container f20cfe17e11643433b2160fcf9ef7c4772414bc0a986f56319bdbbe4f097d0b2: Status 404 returned error can't find the container with id f20cfe17e11643433b2160fcf9ef7c4772414bc0a986f56319bdbbe4f097d0b2 Nov 03 22:19:37.348263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.348239 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fjzr6" Nov 03 22:19:37.348263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.348251 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" event={"ID":"5762840f-90fb-4277-a566-47c3438299d7","Type":"ContainerStarted","Data":"b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3"} Nov 03 22:19:37.348739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.348304 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" event={"ID":"5762840f-90fb-4277-a566-47c3438299d7","Type":"ContainerStarted","Data":"f20cfe17e11643433b2160fcf9ef7c4772414bc0a986f56319bdbbe4f097d0b2"} Nov 03 22:19:37.348739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.348480 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-8l525" Nov 03 22:19:37.397409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.397360 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" podStartSLOduration=1.677879977 podStartE2EDuration="2.397346508s" totalImagesPullingTime="719.466531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:19:37.397027854 +0000 UTC m=+403.800968830" watchObservedRunningTime="2026-11-03 22:19:37.397346508 +0000 UTC m=+403.801287483" Nov 03 22:19:37.422724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.422690 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zkzdg"] Nov 03 22:19:37.422937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.422912 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" podUID="d25b1701-3aca-4d22-a013-0fddfc92ab62" containerName="authorino" containerID="cri-o://d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521" gracePeriod=30 Nov 03 22:19:37.453026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.452994 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8l525"] Nov 03 22:19:37.456619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.456592 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8l525"] Nov 03 22:19:37.509532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.509498 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fjzr6"] Nov 03 22:19:37.512829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.512804 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fjzr6"] Nov 03 22:19:37.657466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.657436 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" Nov 03 22:19:37.731239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.731208 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8347eda6-dc7f-426b-9254-a395f1fa86fa" path="/var/lib/kubelet/pods/8347eda6-dc7f-426b-9254-a395f1fa86fa/volumes" Nov 03 22:19:37.731557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.731541 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ab0df6-fcba-433b-8e25-7298d6c6cc14" path="/var/lib/kubelet/pods/95ab0df6-fcba-433b-8e25-7298d6c6cc14/volumes" Nov 03 22:19:37.756558 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.756524 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25b1701-3aca-4d22-a013-0fddfc92ab62-kube-api-access-f2lpf\" (UniqueName: \"kubernetes.io/projected/d25b1701-3aca-4d22-a013-0fddfc92ab62-kube-api-access-f2lpf\") pod \"d25b1701-3aca-4d22-a013-0fddfc92ab62\" (UID: \"d25b1701-3aca-4d22-a013-0fddfc92ab62\") " Nov 03 22:19:37.758802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.758776 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25b1701-3aca-4d22-a013-0fddfc92ab62-kube-api-access-f2lpf" pod "d25b1701-3aca-4d22-a013-0fddfc92ab62" (UID: "d25b1701-3aca-4d22-a013-0fddfc92ab62"). InnerVolumeSpecName "kube-api-access-f2lpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:19:37.857573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:37.857479 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2lpf\" (UniqueName: \"kubernetes.io/projected/d25b1701-3aca-4d22-a013-0fddfc92ab62-kube-api-access-f2lpf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:19:38.353265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:38.353225 4903 generic.go:356] "Generic (PLEG): container finished" podID="d25b1701-3aca-4d22-a013-0fddfc92ab62" containerID="d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521" exitCode=0 Nov 03 22:19:38.353685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:38.353271 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" Nov 03 22:19:38.353685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:38.353325 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" event={"ID":"d25b1701-3aca-4d22-a013-0fddfc92ab62","Type":"ContainerDied","Data":"d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521"} Nov 03 22:19:38.353685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:38.353365 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zkzdg" event={"ID":"d25b1701-3aca-4d22-a013-0fddfc92ab62","Type":"ContainerDied","Data":"4e90ccaa687f075ee08b7087bc3f799d9215d111b460b63d80edf66e9497151f"} Nov 03 22:19:38.353685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:38.353389 4903 scope.go:122] "RemoveContainer" containerID="d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521" Nov 03 22:19:38.361651 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:38.361636 4903 scope.go:122] "RemoveContainer" containerID="d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521" Nov 03 22:19:38.361924 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:19:38.361906 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521\": container with ID starting with d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521 not found: ID does not exist" containerID="d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521" Nov 03 22:19:38.361970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:38.361933 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521"} err="failed to get container status \"d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521\": rpc error: code = NotFound desc = could not find container \"d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521\": container with ID starting with d14fdd3fcbefdf066d6ba41a5185a1b504ab84512779ee084faad3f6d8fe6521 not found: ID does not exist" Nov 03 22:19:38.398335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:38.398305 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zkzdg"] Nov 03 22:19:38.402363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:38.402339 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zkzdg"] Nov 03 22:19:39.730318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:19:39.730273 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25b1701-3aca-4d22-a013-0fddfc92ab62" path="/var/lib/kubelet/pods/d25b1701-3aca-4d22-a013-0fddfc92ab62/volumes" Nov 03 22:19:48.382947 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:19:48.382907 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:20:18.382939 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:20:18.382895 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:20:48.383371 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:20:48.383288 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:20:49.335592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.335560 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t"] Nov 03 22:20:49.336000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.335976 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d25b1701-3aca-4d22-a013-0fddfc92ab62" containerName="authorino" Nov 03 22:20:49.336000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.335999 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25b1701-3aca-4d22-a013-0fddfc92ab62" containerName="authorino" Nov 03 22:20:49.336261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.336128 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25b1701-3aca-4d22-a013-0fddfc92ab62" containerName="authorino" Nov 03 22:20:49.339261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.339237 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.352089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.352065 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 22:20:49.352196 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.352177 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 22:20:49.361762 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.361741 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lfpms\"" Nov 03 22:20:49.361762 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.361744 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 22:20:49.363068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.363043 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t"] Nov 03 22:20:49.441093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.441063 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.441093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.441100 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc38553-698d-4870-9a3a-cb198e6dc530-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.441752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.441129 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.441752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.441277 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.441752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.441397 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzcl8\" (UniqueName: \"kubernetes.io/projected/dbc38553-698d-4870-9a3a-cb198e6dc530-kube-api-access-dzcl8\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.441752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.441437 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.542399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.542364 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.542721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.542412 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.542721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.542456 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcl8\" (UniqueName: \"kubernetes.io/projected/dbc38553-698d-4870-9a3a-cb198e6dc530-kube-api-access-dzcl8\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.542721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.542481 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.542721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.542535 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.542721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.542568 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc38553-698d-4870-9a3a-cb198e6dc530-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.543077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.542893 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.543077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.542982 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.543157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.543069 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.545040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.545010 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc38553-698d-4870-9a3a-cb198e6dc530-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.545338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.545321 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc38553-698d-4870-9a3a-cb198e6dc530-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.564909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.564881 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzcl8\" (UniqueName: \"kubernetes.io/projected/dbc38553-698d-4870-9a3a-cb198e6dc530-kube-api-access-dzcl8\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t\" (UID: \"dbc38553-698d-4870-9a3a-cb198e6dc530\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.652908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.652886 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:49.946272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:49.946244 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t"] Nov 03 22:20:49.948841 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:20:49.948811 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc38553_698d_4870_9a3a_cb198e6dc530.slice/crio-e4a8b41af26703fa498cffc818ccf232c40e5366bd805f1be0bcae8df841978e WatchSource:0}: Error finding container e4a8b41af26703fa498cffc818ccf232c40e5366bd805f1be0bcae8df841978e: Status 404 returned error can't find the container with id e4a8b41af26703fa498cffc818ccf232c40e5366bd805f1be0bcae8df841978e Nov 03 22:20:50.559321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:50.556664 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:20:50.657796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:50.657762 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" event={"ID":"dbc38553-698d-4870-9a3a-cb198e6dc530","Type":"ContainerStarted","Data":"e4a8b41af26703fa498cffc818ccf232c40e5366bd805f1be0bcae8df841978e"} Nov 03 22:20:51.412650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.412613 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h"] Nov 03 22:20:51.415983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.415961 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.430511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.430482 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 22:20:51.441327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.441283 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h"] Nov 03 22:20:51.463881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.463851 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.464026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.463888 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.464026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.463914 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c282635b-b4c8-40f8-880f-a773dc994c41-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.464127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.464066 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.464176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.464117 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjfd5\" (UniqueName: \"kubernetes.io/projected/c282635b-b4c8-40f8-880f-a773dc994c41-kube-api-access-kjfd5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.464259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.464239 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.568890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.568698 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.568890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.568755 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.568890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.568783 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c282635b-b4c8-40f8-880f-a773dc994c41-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.568890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.568829 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.568890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.568862 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjfd5\" (UniqueName: \"kubernetes.io/projected/c282635b-b4c8-40f8-880f-a773dc994c41-kube-api-access-kjfd5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.569593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.568936 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.569593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.569379 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.569701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.569637 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.570122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.570089 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.572043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.572019 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c282635b-b4c8-40f8-880f-a773dc994c41-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.572399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.572380 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c282635b-b4c8-40f8-880f-a773dc994c41-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.590572 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.590539 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjfd5\" (UniqueName: \"kubernetes.io/projected/c282635b-b4c8-40f8-880f-a773dc994c41-kube-api-access-kjfd5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h\" (UID: \"c282635b-b4c8-40f8-880f-a773dc994c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:51.664557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.664523 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" event={"ID":"dbc38553-698d-4870-9a3a-cb198e6dc530","Type":"ContainerStarted","Data":"9d0d6564ddbd4b4c7c2e7a4017dab34e8b2cd883c57c8987f27c5b3c22cb5bc3"} Nov 03 22:20:51.702965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.702495 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" podStartSLOduration=1.377895898 podStartE2EDuration="2.70247488s" totalImagesPullingTime="1.324578982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:51.701043095 +0000 UTC m=+478.104984085" watchObservedRunningTime="2026-11-03 22:20:51.70247488 +0000 UTC m=+478.106415854" Nov 03 22:20:51.727674 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:51.727635 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:52.023794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.023690 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h"] Nov 03 22:20:52.027067 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:20:52.027038 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc282635b_b4c8_40f8_880f_a773dc994c41.slice/crio-a5dab6addf900f2c34f0c892a17efcfc7df6acf11fd1912f561bc4d9838ecf69 WatchSource:0}: Error finding container a5dab6addf900f2c34f0c892a17efcfc7df6acf11fd1912f561bc4d9838ecf69: Status 404 returned error can't find the container with id a5dab6addf900f2c34f0c892a17efcfc7df6acf11fd1912f561bc4d9838ecf69 Nov 03 22:20:52.653836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.653808 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:52.670312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.670259 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" event={"ID":"c282635b-b4c8-40f8-880f-a773dc994c41","Type":"ContainerStarted","Data":"9846190a1fdc050a25ee4558b576fbe965cdb9d8f1e8421be27533537031fe3c"} Nov 03 22:20:52.670462 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.670326 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" event={"ID":"c282635b-b4c8-40f8-880f-a773dc994c41","Type":"ContainerStarted","Data":"a5dab6addf900f2c34f0c892a17efcfc7df6acf11fd1912f561bc4d9838ecf69"} Nov 03 22:20:52.728002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.727968 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:20:52.790470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.790255 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podStartSLOduration=1.559354537 podStartE2EDuration="1.790233017s" totalImagesPullingTime="230.87848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:52.720165608 +0000 UTC m=+479.124106583" watchObservedRunningTime="2026-11-03 22:20:52.790233017 +0000 UTC m=+479.194173993" Nov 03 22:20:52.791409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.791377 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949"] Nov 03 22:20:52.795289 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.795267 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.811071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.811046 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 22:20:52.819150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.819108 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949"] Nov 03 22:20:52.881935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.881876 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.882131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.882036 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f56bb8-a11b-4b4a-86b0-68f9c535f694-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.882131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.882110 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.882245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.882223 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566bn\" (UniqueName: \"kubernetes.io/projected/29f56bb8-a11b-4b4a-86b0-68f9c535f694-kube-api-access-566bn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.882283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.882255 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.882358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.882350 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.983037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.982937 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566bn\" (UniqueName: \"kubernetes.io/projected/29f56bb8-a11b-4b4a-86b0-68f9c535f694-kube-api-access-566bn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.983037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.982993 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.983276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.983052 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.983276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.983089 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.983276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.983118 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f56bb8-a11b-4b4a-86b0-68f9c535f694-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.983276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.983151 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.983558 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.983528 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.983671 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.983618 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.983751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.983724 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.985455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.985433 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f56bb8-a11b-4b4a-86b0-68f9c535f694-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:52.986042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:52.986018 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f56bb8-a11b-4b4a-86b0-68f9c535f694-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:53.004023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:53.003996 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566bn\" (UniqueName: \"kubernetes.io/projected/29f56bb8-a11b-4b4a-86b0-68f9c535f694-kube-api-access-566bn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bw949\" (UID: \"29f56bb8-a11b-4b4a-86b0-68f9c535f694\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:53.109127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:53.109090 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:53.410895 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:53.410307 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949"] Nov 03 22:20:53.413717 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:20:53.413685 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f56bb8_a11b_4b4a_86b0_68f9c535f694.slice/crio-0fedc9c7f2dfa8cf65ecfd2808387fc08bfe5dffd2adf0754180c1be0f757183 WatchSource:0}: Error finding container 0fedc9c7f2dfa8cf65ecfd2808387fc08bfe5dffd2adf0754180c1be0f757183: Status 404 returned error can't find the container with id 0fedc9c7f2dfa8cf65ecfd2808387fc08bfe5dffd2adf0754180c1be0f757183 Nov 03 22:20:53.678221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:53.678194 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" event={"ID":"29f56bb8-a11b-4b4a-86b0-68f9c535f694","Type":"ContainerStarted","Data":"0fedc9c7f2dfa8cf65ecfd2808387fc08bfe5dffd2adf0754180c1be0f757183"} Nov 03 22:20:54.684967 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:54.684928 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" event={"ID":"29f56bb8-a11b-4b4a-86b0-68f9c535f694","Type":"ContainerStarted","Data":"d0f8829d977c96a221d69e109a02e8d02f9ff4a6811696fb473011497a5b71af"} Nov 03 22:20:54.724094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:54.724037 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" podStartSLOduration=2.489465898 podStartE2EDuration="2.724024601s" totalImagesPullingTime="234.558703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:54.723237314 +0000 UTC m=+481.127178289" watchObservedRunningTime="2026-11-03 22:20:54.724024601 +0000 UTC m=+481.127965576" Nov 03 22:20:55.008788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.008699 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:20:55.109983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.109951 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:20:55.715397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.715353 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q"] Nov 03 22:20:55.719481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.719455 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.742183 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.742117 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 22:20:55.752825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.752797 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q"] Nov 03 22:20:55.809606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.809569 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.809606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.809605 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.809825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.809625 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.809825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.809653 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.809825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.809720 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.809969 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.809872 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvx8v\" (UniqueName: \"kubernetes.io/projected/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-kube-api-access-xvx8v\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.910886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.910844 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.911070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.910899 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.911070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.910952 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.911070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.910989 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.911070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.911014 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.911277 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.911098 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvx8v\" (UniqueName: \"kubernetes.io/projected/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-kube-api-access-xvx8v\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.911396 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.911373 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.911465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.911383 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.912037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.912006 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.913708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.913685 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.914406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.914383 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:55.933697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:55.933664 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvx8v\" (UniqueName: \"kubernetes.io/projected/a8b9b4d3-1dde-4142-84e9-3391d26a3b1c-kube-api-access-xvx8v\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2z58q\" (UID: \"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:56.051009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:56.050908 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:56.354149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:56.354120 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q"] Nov 03 22:20:56.356763 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:20:56.356733 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b9b4d3_1dde_4142_84e9_3391d26a3b1c.slice/crio-0500e8db36fbe026b7a9f783cbae49781f83e72e90cc26291bf3ced673ca0f91 WatchSource:0}: Error finding container 0500e8db36fbe026b7a9f783cbae49781f83e72e90cc26291bf3ced673ca0f91: Status 404 returned error can't find the container with id 0500e8db36fbe026b7a9f783cbae49781f83e72e90cc26291bf3ced673ca0f91 Nov 03 22:20:56.696337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:56.696268 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" event={"ID":"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c","Type":"ContainerStarted","Data":"a119341df0199c862a2b8026acb96126eb1453ba14c977731e2a19503e39f9e4"} Nov 03 22:20:56.696527 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:56.696353 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" event={"ID":"a8b9b4d3-1dde-4142-84e9-3391d26a3b1c","Type":"ContainerStarted","Data":"0500e8db36fbe026b7a9f783cbae49781f83e72e90cc26291bf3ced673ca0f91"} Nov 03 22:20:56.734406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:56.734353 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" podStartSLOduration=1.502177541 podStartE2EDuration="1.734339097s" totalImagesPullingTime="232.161556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:20:56.733429572 +0000 UTC m=+483.137370559" watchObservedRunningTime="2026-11-03 22:20:56.734339097 +0000 UTC m=+483.138280071" Nov 03 22:20:57.051714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:57.051681 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:20:59.653487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:59.653446 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:59.666195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:59.666166 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:20:59.720533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:20:59.720500 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t" Nov 03 22:21:00.527336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.527288 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7"] Nov 03 22:21:00.531065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.531043 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.553597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.553570 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 22:21:00.555693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.555665 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7"] Nov 03 22:21:00.662225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.662187 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d72039-62c4-4d01-85f9-4e65d431a4a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.662225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.662230 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99hjq\" (UniqueName: \"kubernetes.io/projected/c9d72039-62c4-4d01-85f9-4e65d431a4a8-kube-api-access-99hjq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.662726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.662260 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.662726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.662386 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.662726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.662432 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.662726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.662457 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.763619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.763585 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.763819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.763669 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d72039-62c4-4d01-85f9-4e65d431a4a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.763819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.763700 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99hjq\" (UniqueName: \"kubernetes.io/projected/c9d72039-62c4-4d01-85f9-4e65d431a4a8-kube-api-access-99hjq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.763819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.763734 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.763819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.763788 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.764032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.763831 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.764204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.764177 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.764427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.764403 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.764704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.764684 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.766465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.766435 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d72039-62c4-4d01-85f9-4e65d431a4a8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.766800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.766777 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d72039-62c4-4d01-85f9-4e65d431a4a8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.785805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.785743 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99hjq\" (UniqueName: \"kubernetes.io/projected/c9d72039-62c4-4d01-85f9-4e65d431a4a8-kube-api-access-99hjq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7\" (UID: \"c9d72039-62c4-4d01-85f9-4e65d431a4a8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:00.847853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:00.847812 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:01.150473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:01.150246 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7"] Nov 03 22:21:01.153975 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:21:01.153950 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d72039_62c4_4d01_85f9_4e65d431a4a8.slice/crio-7353b43d51859819e5b82e81978fcfe4626780d40638a32b3962ad372e24f8d1 WatchSource:0}: Error finding container 7353b43d51859819e5b82e81978fcfe4626780d40638a32b3962ad372e24f8d1: Status 404 returned error can't find the container with id 7353b43d51859819e5b82e81978fcfe4626780d40638a32b3962ad372e24f8d1 Nov 03 22:21:01.718802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:01.718765 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" event={"ID":"c9d72039-62c4-4d01-85f9-4e65d431a4a8","Type":"ContainerStarted","Data":"4b806559dfefc28166d517d4a7a4a7c1d0ba02d9c43ac6279097f065a7c2f76f"} Nov 03 22:21:01.719338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:01.718838 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" event={"ID":"c9d72039-62c4-4d01-85f9-4e65d431a4a8","Type":"ContainerStarted","Data":"7353b43d51859819e5b82e81978fcfe4626780d40638a32b3962ad372e24f8d1"} Nov 03 22:21:01.730689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:01.730660 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:21:01.741457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:01.741427 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:21:01.757930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:01.757751 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" podStartSLOduration=1.573490499 podStartE2EDuration="1.757733439s" totalImagesPullingTime="184.24294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:01.757135935 +0000 UTC m=+488.161076926" watchObservedRunningTime="2026-11-03 22:21:01.757733439 +0000 UTC m=+488.161674415" Nov 03 22:21:01.848195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:01.848160 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:02.733913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:02.733881 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:21:03.109889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:03.109804 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:21:03.122345 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:03.122319 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:21:03.721404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:03.721357 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:21:03.741028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:03.741000 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bw949" Nov 03 22:21:06.052172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:06.052069 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:21:06.071776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:06.071747 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:21:06.752813 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:06.752782 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" Nov 03 22:21:10.848309 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:10.848266 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:10.860657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:10.860634 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:11.772845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:11.772814 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" Nov 03 22:21:13.224128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.224094 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9"] Nov 03 22:21:13.227924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.227902 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.243843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.243815 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 22:21:13.250707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.250682 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9d7jc\"" Nov 03 22:21:13.253094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.253070 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9"] Nov 03 22:21:13.285303 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.285254 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.285492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.285315 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.285492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.285425 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.285492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.285475 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.285645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.285513 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.285645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.285555 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvxg\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-kube-api-access-wmvxg\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.285743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.285637 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.285743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.285694 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.285857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.285790 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387050 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387093 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387114 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387137 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387171 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvxg\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-kube-api-access-wmvxg\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387221 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387240 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387280 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387352 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387519 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387616 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387761 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.387961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.387874 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.388080 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.388061 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.389628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.389607 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.389980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.389962 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.408408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.408380 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.408503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.408428 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvxg\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-kube-api-access-wmvxg\") pod \"maas-default-gateway-openshift-default-7df854bf67-qhqx9\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.540972 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.540890 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:13.851982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:13.851955 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9"] Nov 03 22:21:13.854543 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:21:13.854514 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4e44ab_74c0_48d1_8d22_4c9d683a077a.slice/crio-ec005ba060531a0ce177325704aa602ebd8fc3b70b3d10beb4473078c0b599e2 WatchSource:0}: Error finding container ec005ba060531a0ce177325704aa602ebd8fc3b70b3d10beb4473078c0b599e2: Status 404 returned error can't find the container with id ec005ba060531a0ce177325704aa602ebd8fc3b70b3d10beb4473078c0b599e2 Nov 03 22:21:14.777933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:14.777884 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" event={"ID":"6d4e44ab-74c0-48d1-8d22-4c9d683a077a","Type":"ContainerStarted","Data":"ec005ba060531a0ce177325704aa602ebd8fc3b70b3d10beb4473078c0b599e2"} Nov 03 22:21:16.014002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.013955 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59cc64bfd5-wtjfp"] Nov 03 22:21:16.021371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.021349 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" Nov 03 22:21:16.034152 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.034121 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59cc64bfd5-wtjfp"] Nov 03 22:21:16.113206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.113162 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f374375e-68d7-4898-9a67-c4bb3b1be125-tls-cert\") pod \"authorino-59cc64bfd5-wtjfp\" (UID: \"f374375e-68d7-4898-9a67-c4bb3b1be125\") " pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" Nov 03 22:21:16.113420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.113222 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcw4f\" (UniqueName: \"kubernetes.io/projected/f374375e-68d7-4898-9a67-c4bb3b1be125-kube-api-access-wcw4f\") pod \"authorino-59cc64bfd5-wtjfp\" (UID: \"f374375e-68d7-4898-9a67-c4bb3b1be125\") " pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" Nov 03 22:21:16.214494 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.214456 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f374375e-68d7-4898-9a67-c4bb3b1be125-tls-cert\") pod \"authorino-59cc64bfd5-wtjfp\" (UID: \"f374375e-68d7-4898-9a67-c4bb3b1be125\") " pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" Nov 03 22:21:16.214701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.214506 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcw4f\" (UniqueName: \"kubernetes.io/projected/f374375e-68d7-4898-9a67-c4bb3b1be125-kube-api-access-wcw4f\") pod \"authorino-59cc64bfd5-wtjfp\" (UID: \"f374375e-68d7-4898-9a67-c4bb3b1be125\") " pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" Nov 03 22:21:16.217117 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.217087 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f374375e-68d7-4898-9a67-c4bb3b1be125-tls-cert\") pod \"authorino-59cc64bfd5-wtjfp\" (UID: \"f374375e-68d7-4898-9a67-c4bb3b1be125\") " pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" Nov 03 22:21:16.235901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.235869 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcw4f\" (UniqueName: \"kubernetes.io/projected/f374375e-68d7-4898-9a67-c4bb3b1be125-kube-api-access-wcw4f\") pod \"authorino-59cc64bfd5-wtjfp\" (UID: \"f374375e-68d7-4898-9a67-c4bb3b1be125\") " pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" Nov 03 22:21:16.335264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.335178 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" Nov 03 22:21:16.521442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.521413 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:21:16.521525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.521480 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:21:16.521525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.521510 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:21:16.641070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.640994 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59cc64bfd5-wtjfp"] Nov 03 22:21:16.644200 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:21:16.644177 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf374375e_68d7_4898_9a67_c4bb3b1be125.slice/crio-501f0922a0c960c74e63c03246d589fb209edc7fe5ecbe6adce7053106b3b745 WatchSource:0}: Error finding container 501f0922a0c960c74e63c03246d589fb209edc7fe5ecbe6adce7053106b3b745: Status 404 returned error can't find the container with id 501f0922a0c960c74e63c03246d589fb209edc7fe5ecbe6adce7053106b3b745 Nov 03 22:21:16.787857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.787824 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" event={"ID":"f374375e-68d7-4898-9a67-c4bb3b1be125","Type":"ContainerStarted","Data":"501f0922a0c960c74e63c03246d589fb209edc7fe5ecbe6adce7053106b3b745"} Nov 03 22:21:16.789703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.789673 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" event={"ID":"6d4e44ab-74c0-48d1-8d22-4c9d683a077a","Type":"ContainerStarted","Data":"e424f79d1bf43a378361a952cf40f646f82fea5ced2cc7b142e432035df7d97f"} Nov 03 22:21:16.830575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:16.830523 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podStartSLOduration=1.166379074 podStartE2EDuration="3.830506034s" totalImagesPullingTime="2.66412696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:16.829378522 +0000 UTC m=+503.233319499" watchObservedRunningTime="2026-11-03 22:21:16.830506034 +0000 UTC m=+503.234446988" Nov 03 22:21:17.548316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:17.545618 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:17.565381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:17.560421 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:21:17.565381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:17.560487 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 03 22:21:17.796692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:17.796651 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" event={"ID":"f374375e-68d7-4898-9a67-c4bb3b1be125","Type":"ContainerStarted","Data":"fd347a02787e544f5d34ead672e3a325d6667b2ab1aaee7bede121c25aa92205"} Nov 03 22:21:17.834386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:17.833947 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-59cc64bfd5-wtjfp" podStartSLOduration=2.386743212 podStartE2EDuration="2.833926172s" totalImagesPullingTime="447.18296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:17.830565467 +0000 UTC m=+504.234506443" watchObservedRunningTime="2026-11-03 22:21:17.833926172 +0000 UTC m=+504.237867149" Nov 03 22:21:17.873318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:17.871757 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bf5dfd866-kkskx"] Nov 03 22:21:17.873318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:17.871985 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" podUID="5762840f-90fb-4277-a566-47c3438299d7" containerName="authorino" containerID="cri-o://b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3" gracePeriod=30 Nov 03 22:21:18.199602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.199575 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:21:18.339471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.339437 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5762840f-90fb-4277-a566-47c3438299d7-tls-cert\" (UniqueName: \"kubernetes.io/secret/5762840f-90fb-4277-a566-47c3438299d7-tls-cert\") pod \"5762840f-90fb-4277-a566-47c3438299d7\" (UID: \"5762840f-90fb-4277-a566-47c3438299d7\") " Nov 03 22:21:18.339645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.339522 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5762840f-90fb-4277-a566-47c3438299d7-kube-api-access-95f2n\" (UniqueName: \"kubernetes.io/projected/5762840f-90fb-4277-a566-47c3438299d7-kube-api-access-95f2n\") pod \"5762840f-90fb-4277-a566-47c3438299d7\" (UID: \"5762840f-90fb-4277-a566-47c3438299d7\") " Nov 03 22:21:18.369118 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.368470 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5762840f-90fb-4277-a566-47c3438299d7-kube-api-access-95f2n" pod "5762840f-90fb-4277-a566-47c3438299d7" (UID: "5762840f-90fb-4277-a566-47c3438299d7"). InnerVolumeSpecName "kube-api-access-95f2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:21:18.380241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.380202 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5762840f-90fb-4277-a566-47c3438299d7-tls-cert" pod "5762840f-90fb-4277-a566-47c3438299d7" (UID: "5762840f-90fb-4277-a566-47c3438299d7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:21:18.412937 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:21:18.412905 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:21:18.440529 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.440414 4903 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5762840f-90fb-4277-a566-47c3438299d7-tls-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:21:18.440529 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.440449 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95f2n\" (UniqueName: \"kubernetes.io/projected/5762840f-90fb-4277-a566-47c3438299d7-kube-api-access-95f2n\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:21:18.565336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.561421 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:21:18.565336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.561485 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 03 22:21:18.802982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.802897 4903 generic.go:356] "Generic (PLEG): container finished" podID="5762840f-90fb-4277-a566-47c3438299d7" containerID="b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3" exitCode=0 Nov 03 22:21:18.802982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.802937 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" Nov 03 22:21:18.803180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.802975 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" event={"ID":"5762840f-90fb-4277-a566-47c3438299d7","Type":"ContainerDied","Data":"b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3"} Nov 03 22:21:18.803180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.803022 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf5dfd866-kkskx" event={"ID":"5762840f-90fb-4277-a566-47c3438299d7","Type":"ContainerDied","Data":"f20cfe17e11643433b2160fcf9ef7c4772414bc0a986f56319bdbbe4f097d0b2"} Nov 03 22:21:18.803180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.803039 4903 scope.go:122] "RemoveContainer" containerID="b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3" Nov 03 22:21:18.813077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.813057 4903 scope.go:122] "RemoveContainer" containerID="b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3" Nov 03 22:21:18.813338 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:21:18.813318 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3\": container with ID starting with b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3 not found: ID does not exist" containerID="b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3" Nov 03 22:21:18.813440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.813346 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3"} err="failed to get container status \"b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3\": rpc error: code = NotFound desc = could not find container \"b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3\": container with ID starting with b54e9bd097b9deb302cbdd7fef3443a6b79ababac475169ad10ef8124b8310e3 not found: ID does not exist" Nov 03 22:21:18.859319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.849855 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bf5dfd866-kkskx"] Nov 03 22:21:18.868316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:18.865316 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6bf5dfd866-kkskx"] Nov 03 22:21:19.545321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:19.544723 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:21:19.545321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:19.544787 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 03 22:21:19.731660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:19.731623 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5762840f-90fb-4277-a566-47c3438299d7" path="/var/lib/kubelet/pods/5762840f-90fb-4277-a566-47c3438299d7/volumes" Nov 03 22:21:20.549625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:20.549583 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:21:20.549907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:20.549875 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 03 22:21:21.546726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:21.546684 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:21:21.547282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:21.547247 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 03 22:21:22.541752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.541707 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:21:22.541952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.541780 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 03 22:21:22.722318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.721799 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-bnqlr"] Nov 03 22:21:22.722800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.722372 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5762840f-90fb-4277-a566-47c3438299d7" containerName="authorino" Nov 03 22:21:22.722800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.722392 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5762840f-90fb-4277-a566-47c3438299d7" containerName="authorino" Nov 03 22:21:22.722800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.722512 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5762840f-90fb-4277-a566-47c3438299d7" containerName="authorino" Nov 03 22:21:22.730674 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.730632 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:22.735971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.735944 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-bnqlr"] Nov 03 22:21:22.744091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.744066 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbplq\" (UniqueName: \"kubernetes.io/projected/76f24aa5-5e88-488f-90e3-252e1e114b39-kube-api-access-vbplq\") pod \"payload-processing-7bb9489676-bnqlr\" (UID: \"76f24aa5-5e88-488f-90e3-252e1e114b39\") " pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:22.746835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.745018 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 22:21:22.753405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.753375 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f24aa5-5e88-488f-90e3-252e1e114b39-plugins-config-volume\") pod \"payload-processing-7bb9489676-bnqlr\" (UID: \"76f24aa5-5e88-488f-90e3-252e1e114b39\") " pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:22.754553 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.754520 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5tcms\"" Nov 03 22:21:22.779673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.779642 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4"] Nov 03 22:21:22.785597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.785569 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:22.796228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.794702 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4"] Nov 03 22:21:22.854567 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.854533 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9af6c0-52ea-4612-8d1f-b3d4138c8f32-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-wnlt4\" (UID: \"5d9af6c0-52ea-4612-8d1f-b3d4138c8f32\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:22.854751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.854574 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f24aa5-5e88-488f-90e3-252e1e114b39-plugins-config-volume\") pod \"payload-processing-7bb9489676-bnqlr\" (UID: \"76f24aa5-5e88-488f-90e3-252e1e114b39\") " pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:22.854751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.854662 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbplq\" (UniqueName: \"kubernetes.io/projected/76f24aa5-5e88-488f-90e3-252e1e114b39-kube-api-access-vbplq\") pod \"payload-processing-7bb9489676-bnqlr\" (UID: \"76f24aa5-5e88-488f-90e3-252e1e114b39\") " pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:22.854751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.854747 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7475\" (UniqueName: \"kubernetes.io/projected/5d9af6c0-52ea-4612-8d1f-b3d4138c8f32-kube-api-access-v7475\") pod \"payload-pre-processing-7b98bb678c-wnlt4\" (UID: \"5d9af6c0-52ea-4612-8d1f-b3d4138c8f32\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:22.855433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.855409 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f24aa5-5e88-488f-90e3-252e1e114b39-plugins-config-volume\") pod \"payload-processing-7bb9489676-bnqlr\" (UID: \"76f24aa5-5e88-488f-90e3-252e1e114b39\") " pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:22.875762 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.875736 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbplq\" (UniqueName: \"kubernetes.io/projected/76f24aa5-5e88-488f-90e3-252e1e114b39-kube-api-access-vbplq\") pod \"payload-processing-7bb9489676-bnqlr\" (UID: \"76f24aa5-5e88-488f-90e3-252e1e114b39\") " pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:22.955469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.955439 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7475\" (UniqueName: \"kubernetes.io/projected/5d9af6c0-52ea-4612-8d1f-b3d4138c8f32-kube-api-access-v7475\") pod \"payload-pre-processing-7b98bb678c-wnlt4\" (UID: \"5d9af6c0-52ea-4612-8d1f-b3d4138c8f32\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:22.955645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.955489 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9af6c0-52ea-4612-8d1f-b3d4138c8f32-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-wnlt4\" (UID: \"5d9af6c0-52ea-4612-8d1f-b3d4138c8f32\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:22.956578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.956551 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9af6c0-52ea-4612-8d1f-b3d4138c8f32-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-wnlt4\" (UID: \"5d9af6c0-52ea-4612-8d1f-b3d4138c8f32\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:22.977068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:22.977042 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7475\" (UniqueName: \"kubernetes.io/projected/5d9af6c0-52ea-4612-8d1f-b3d4138c8f32-kube-api-access-v7475\") pod \"payload-pre-processing-7b98bb678c-wnlt4\" (UID: \"5d9af6c0-52ea-4612-8d1f-b3d4138c8f32\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:23.049139 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:23.049052 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:23.102306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:23.101952 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:23.358574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:23.358543 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-bnqlr"] Nov 03 22:21:23.360789 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:21:23.360753 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f24aa5_5e88_488f_90e3_252e1e114b39.slice/crio-1a7421c4230b9cc5c0b44e21cea4aadd408b777c9509cd6f8b9fbc295e28db61 WatchSource:0}: Error finding container 1a7421c4230b9cc5c0b44e21cea4aadd408b777c9509cd6f8b9fbc295e28db61: Status 404 returned error can't find the container with id 1a7421c4230b9cc5c0b44e21cea4aadd408b777c9509cd6f8b9fbc295e28db61 Nov 03 22:21:23.435450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:23.435371 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4"] Nov 03 22:21:23.438441 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:21:23.438407 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9af6c0_52ea_4612_8d1f_b3d4138c8f32.slice/crio-11b32a1a335ef43990cdbd24c47c5c21082ba523132a17dab906957274c3a480 WatchSource:0}: Error finding container 11b32a1a335ef43990cdbd24c47c5c21082ba523132a17dab906957274c3a480: Status 404 returned error can't find the container with id 11b32a1a335ef43990cdbd24c47c5c21082ba523132a17dab906957274c3a480 Nov 03 22:21:23.542323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:23.542275 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 03 22:21:23.542501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:23.542345 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 03 22:21:23.845576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:23.845525 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" event={"ID":"5d9af6c0-52ea-4612-8d1f-b3d4138c8f32","Type":"ContainerStarted","Data":"11b32a1a335ef43990cdbd24c47c5c21082ba523132a17dab906957274c3a480"} Nov 03 22:21:23.847531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:23.847498 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" event={"ID":"76f24aa5-5e88-488f-90e3-252e1e114b39","Type":"ContainerStarted","Data":"1a7421c4230b9cc5c0b44e21cea4aadd408b777c9509cd6f8b9fbc295e28db61"} Nov 03 22:21:25.553003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:25.552963 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:21:25.553531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:25.553019 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:21:26.554359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:26.554325 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:21:26.554707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:26.554388 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:21:26.876574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:26.876479 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" event={"ID":"5d9af6c0-52ea-4612-8d1f-b3d4138c8f32","Type":"ContainerStarted","Data":"bcb961cc7a0e73d374c1b0a23f64be3313a30b46816005aa700f306d011d969e"} Nov 03 22:21:26.876749 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:26.876592 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:26.878527 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:26.878501 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" event={"ID":"76f24aa5-5e88-488f-90e3-252e1e114b39","Type":"ContainerStarted","Data":"bdc200357b3545e3728808567c51d09afcc18242865d3959ce9d30aefc6bbf3c"} Nov 03 22:21:26.878699 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:26.878596 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:26.915416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:26.915363 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" podStartSLOduration=1.9117744939999999 podStartE2EDuration="4.915347651s" totalImagesPullingTime="3.003573157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:26.915330421 +0000 UTC m=+513.319271393" watchObservedRunningTime="2026-11-03 22:21:26.915347651 +0000 UTC m=+513.319288627" Nov 03 22:21:26.952400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:26.952339 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" podStartSLOduration=1.8368612880000001 podStartE2EDuration="4.952318584s" totalImagesPullingTime="3.115457296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:26.950752466 +0000 UTC m=+513.354693441" watchObservedRunningTime="2026-11-03 22:21:26.952318584 +0000 UTC m=+513.356259561" Nov 03 22:21:27.596443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:27.596403 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:21:27.596950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:27.596460 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:21:27.872123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:27.872033 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:27.872419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:27.872400 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:27.873378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:27.873354 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:21:28.269918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.269881 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd"] Nov 03 22:21:28.273360 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.273337 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.287220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.287197 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx"] Nov 03 22:21:28.290856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.290838 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.297908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.297883 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd"] Nov 03 22:21:28.312672 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.312640 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.312817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.312684 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534555a5-ce83-4e60-a28a-b410824419ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.312817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.312752 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558mc\" (UniqueName: \"kubernetes.io/projected/534555a5-ce83-4e60-a28a-b410824419ac-kube-api-access-558mc\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.312855 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.312943 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534555a5-ce83-4e60-a28a-b410824419ac-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313036 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313066 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313119 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8q4\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-kube-api-access-9x8q4\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313159 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313205 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313321 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313522 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313550 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.313591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313576 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.314243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313715 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.314243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313753 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.314243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313794 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.314243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.313818 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534555a5-ce83-4e60-a28a-b410824419ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.327998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.327968 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx"] Nov 03 22:21:28.414765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.414723 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534555a5-ce83-4e60-a28a-b410824419ac-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.414765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.414770 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.414888 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.414927 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8q4\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-kube-api-access-9x8q4\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.414961 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.414982 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415002 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415024 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415044 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415067 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415150 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415186 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415202 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415229 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415255 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534555a5-ce83-4e60-a28a-b410824419ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415326 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415352 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534555a5-ce83-4e60-a28a-b410824419ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415376 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-558mc\" (UniqueName: \"kubernetes.io/projected/534555a5-ce83-4e60-a28a-b410824419ac-kube-api-access-558mc\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415405 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415416 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415418 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415564 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.415891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415736 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.415891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415796 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.416136 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.415980 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.416136 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.416111 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.416229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.416178 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.416424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.416402 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534555a5-ce83-4e60-a28a-b410824419ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.417587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.417555 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534555a5-ce83-4e60-a28a-b410824419ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.417750 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.417730 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.418101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.418085 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534555a5-ce83-4e60-a28a-b410824419ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.418234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.418218 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.436599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.436572 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.436599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.436591 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534555a5-ce83-4e60-a28a-b410824419ac-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.436776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.436761 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8q4\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-kube-api-access-9x8q4\") pod \"maas-default-gateway-openshift-default-7df854bf67-2cwvd\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.436920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.436896 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-558mc\" (UniqueName: \"kubernetes.io/projected/534555a5-ce83-4e60-a28a-b410824419ac-kube-api-access-558mc\") pod \"maas-default-gateway-openshift-default-7df854bf67-xt8wx\" (UID: \"534555a5-ce83-4e60-a28a-b410824419ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.587040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.586953 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:28.606523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.606500 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:28.913509 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.913483 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd"] Nov 03 22:21:28.915835 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:21:28.915806 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1a292c_ffde_4ae7_af34_d84fb47c6021.slice/crio-db43d869040c9bdaa36e6638ca605e96d94cc9377464a1fcd8a677287c32a3ff WatchSource:0}: Error finding container db43d869040c9bdaa36e6638ca605e96d94cc9377464a1fcd8a677287c32a3ff: Status 404 returned error can't find the container with id db43d869040c9bdaa36e6638ca605e96d94cc9377464a1fcd8a677287c32a3ff Nov 03 22:21:28.917921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.917894 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:21:28.917976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.917956 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:21:28.918013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.917985 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:21:28.923633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.923610 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx"] Nov 03 22:21:28.928330 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:21:28.928306 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534555a5_ce83_4e60_a28a_b410824419ac.slice/crio-8caf70518c1e880da111648622a6589b37f2aaca8b4c0ac915590caa354f0aca WatchSource:0}: Error finding container 8caf70518c1e880da111648622a6589b37f2aaca8b4c0ac915590caa354f0aca: Status 404 returned error can't find the container with id 8caf70518c1e880da111648622a6589b37f2aaca8b4c0ac915590caa354f0aca Nov 03 22:21:28.930696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.930671 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:21:28.930804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.930723 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:21:28.930804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:28.930756 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:21:29.896738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:29.896456 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" event={"ID":"8c1a292c-ffde-4ae7-af34-d84fb47c6021","Type":"ContainerStarted","Data":"ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c"} Nov 03 22:21:29.897242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:29.896789 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" event={"ID":"8c1a292c-ffde-4ae7-af34-d84fb47c6021","Type":"ContainerStarted","Data":"db43d869040c9bdaa36e6638ca605e96d94cc9377464a1fcd8a677287c32a3ff"} Nov 03 22:21:29.898175 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:29.898151 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" event={"ID":"534555a5-ce83-4e60-a28a-b410824419ac","Type":"ContainerStarted","Data":"a327fd91eb506b2d06d4c7f5d56b8ad5f655adeed733bbc09b9fcbb73401d425"} Nov 03 22:21:29.898283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:29.898196 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" event={"ID":"534555a5-ce83-4e60-a28a-b410824419ac","Type":"ContainerStarted","Data":"8caf70518c1e880da111648622a6589b37f2aaca8b4c0ac915590caa354f0aca"} Nov 03 22:21:29.967315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:29.966373 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podStartSLOduration=1.966338361 podStartE2EDuration="1.966338361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:29.965633579 +0000 UTC m=+516.369574555" watchObservedRunningTime="2026-11-03 22:21:29.966338361 +0000 UTC m=+516.370279337" Nov 03 22:21:30.009267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:30.009211 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" podStartSLOduration=2.009196188 podStartE2EDuration="2.009196188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:21:30.008492489 +0000 UTC m=+516.412433464" watchObservedRunningTime="2026-11-03 22:21:30.009196188 +0000 UTC m=+516.413137162" Nov 03 22:21:30.588688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:30.588656 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:30.591157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:30.591126 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2cwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 22:21:30.591401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:30.591366 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 22:21:30.607186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:30.607165 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:30.608095 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:30.608075 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xt8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 22:21:30.608180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:30.608116 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" podUID="534555a5-ce83-4e60-a28a-b410824419ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 22:21:31.590319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:31.589764 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2cwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 22:21:31.590319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:31.589828 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 22:21:31.607579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:31.607547 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xt8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 22:21:31.607817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:31.607784 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" podUID="534555a5-ce83-4e60-a28a-b410824419ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 22:21:32.587799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:32.587757 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2cwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 22:21:32.588071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:32.588039 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 22:21:32.607408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:32.607376 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xt8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 22:21:32.607860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:32.607837 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" podUID="534555a5-ce83-4e60-a28a-b410824419ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 22:21:33.588318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:33.587787 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2cwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 22:21:33.588318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:33.587860 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 22:21:33.607759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:33.607726 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xt8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 22:21:33.608133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:33.607784 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" podUID="534555a5-ce83-4e60-a28a-b410824419ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 22:21:34.587809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:34.587764 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2cwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 22:21:34.587994 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:34.587840 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 22:21:34.606989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:34.606957 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xt8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 22:21:34.607142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:34.607013 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" podUID="534555a5-ce83-4e60-a28a-b410824419ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 22:21:35.588150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:35.588103 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2cwvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 22:21:35.588603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:35.588180 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 22:21:35.607595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:35.607560 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xt8wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 22:21:35.607746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:35.607616 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" podUID="534555a5-ce83-4e60-a28a-b410824419ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 22:21:37.186833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:37.186796 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:37.187461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:37.187433 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:37.188373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:37.188353 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" Nov 03 22:21:37.237860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:37.237774 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:37.238424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:37.238340 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:37.239311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:37.239273 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:21:37.887470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:37.887444 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" Nov 03 22:21:37.887617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:21:37.887503 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-wnlt4" Nov 03 22:21:48.384347 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:21:48.384309 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:22:12.347187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.347150 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk"] Nov 03 22:22:12.350664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.350646 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.366673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.366638 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk"] Nov 03 22:22:12.374092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.374063 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-2d619f-openshift-default-dockercfg-2f4cd\"" Nov 03 22:22:12.457011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.456978 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.457172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.457024 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf9ce64-ce17-4f22-9a81-b038a7987369-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.457172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.457095 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.457172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.457121 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nzsn\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-kube-api-access-4nzsn\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.457172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.457145 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.457172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.457168 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.457366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.457314 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.457366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.457352 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.457428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.457414 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.473501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.473270 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn"] Nov 03 22:22:12.477047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.477026 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.494969 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.494948 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-434ce3-openshift-default-dockercfg-4lnm5\"" Nov 03 22:22:12.500243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.500219 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn"] Nov 03 22:22:12.558203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558170 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.558203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558207 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.558428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558232 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.558428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558331 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.558428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558397 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.558528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558455 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.558528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558489 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf9ce64-ce17-4f22-9a81-b038a7987369-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.558528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558519 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.558617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558548 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.558617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558578 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nzsn\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-kube-api-access-4nzsn\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.558679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558610 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xgr\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-kube-api-access-g2xgr\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.558679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558648 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.558761 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558674 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.558761 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558679 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.558873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558852 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.558928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558849 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.558976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558922 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.558976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.558966 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.559087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.559007 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.559087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.559056 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.559190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.559085 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf9ce64-ce17-4f22-9a81-b038a7987369-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.559190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.559116 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.559407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.559369 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.561101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.561081 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.561333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.561313 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.581829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.581803 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.581951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.581884 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nzsn\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-kube-api-access-4nzsn\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.660329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660290 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660341 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660374 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660392 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660419 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660444 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660480 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xgr\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-kube-api-access-g2xgr\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660518 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660592 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660733 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.660785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.660743 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.661044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.661003 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.661166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.661105 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.661524 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.661501 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.662889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.662859 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.663068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.663051 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.663889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.663874 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:12.682619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.682593 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.682756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.682737 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xgr\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-kube-api-access-g2xgr\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.794157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.794124 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:12.980592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.980568 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk"] Nov 03 22:22:12.983251 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:12.983220 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf9ce64_ce17_4f22_9a81_b038a7987369.slice/crio-d02f8860cb8a8130e67abb8a9701f4b31b06c1a80a0369edaba027e099a82d2b WatchSource:0}: Error finding container d02f8860cb8a8130e67abb8a9701f4b31b06c1a80a0369edaba027e099a82d2b: Status 404 returned error can't find the container with id d02f8860cb8a8130e67abb8a9701f4b31b06c1a80a0369edaba027e099a82d2b Nov 03 22:22:12.985401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.985366 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:12.985481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.985433 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:12.985481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:12.985468 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:13.099264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:13.099234 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn"] Nov 03 22:22:13.101703 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:13.101672 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849e7c9f_8596_41a4_b9ed_e7472cecadbb.slice/crio-766e707f049ee9c320a3fae86e8b3f8735a1cb27c5025b94d64528aec6849f11 WatchSource:0}: Error finding container 766e707f049ee9c320a3fae86e8b3f8735a1cb27c5025b94d64528aec6849f11: Status 404 returned error can't find the container with id 766e707f049ee9c320a3fae86e8b3f8735a1cb27c5025b94d64528aec6849f11 Nov 03 22:22:13.104858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:13.104824 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:13.105046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:13.105025 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:13.105185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:13.105167 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:13.107721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:13.107697 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" event={"ID":"acf9ce64-ce17-4f22-9a81-b038a7987369","Type":"ContainerStarted","Data":"70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955"} Nov 03 22:22:13.108089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:13.108060 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" event={"ID":"acf9ce64-ce17-4f22-9a81-b038a7987369","Type":"ContainerStarted","Data":"d02f8860cb8a8130e67abb8a9701f4b31b06c1a80a0369edaba027e099a82d2b"} Nov 03 22:22:13.212228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:13.212169 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" podStartSLOduration=1.212150525 podStartE2EDuration="1.212150525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:13.209879036 +0000 UTC m=+559.613820011" watchObservedRunningTime="2026-11-03 22:22:13.212150525 +0000 UTC m=+559.616091502" Nov 03 22:22:13.664533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:13.664500 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:14.114274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:14.114121 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" event={"ID":"849e7c9f-8596-41a4-b9ed-e7472cecadbb","Type":"ContainerStarted","Data":"9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5"} Nov 03 22:22:14.114274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:14.114187 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" event={"ID":"849e7c9f-8596-41a4-b9ed-e7472cecadbb","Type":"ContainerStarted","Data":"766e707f049ee9c320a3fae86e8b3f8735a1cb27c5025b94d64528aec6849f11"} Nov 03 22:22:14.164630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:14.164583 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" podStartSLOduration=2.1645713779999998 podStartE2EDuration="2.164571378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:14.164236548 +0000 UTC m=+560.568177524" watchObservedRunningTime="2026-11-03 22:22:14.164571378 +0000 UTC m=+560.568512353" Nov 03 22:22:14.670226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:14.670193 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:14.795346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:14.795284 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:14.800523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:14.800498 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:15.118678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:15.118601 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:15.118678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:15.118656 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:15.119514 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:15.119493 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:15.119671 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:15.119656 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:18.043206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.043166 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv"] Nov 03 22:22:18.046739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.046723 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:18.059922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.059899 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-50d738\"" Nov 03 22:22:18.070080 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.070060 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-50d738-dockercfg-4n9f7\"" Nov 03 22:22:18.072116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.072093 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv"] Nov 03 22:22:18.104971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.104940 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf"] Nov 03 22:22:18.110706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.110684 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:18.114386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.114363 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvkx\" (UniqueName: \"kubernetes.io/projected/58aedb13-1685-4dfb-a138-af48cd8e52a9-kube-api-access-5rvkx\") pod \"payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv\" (UID: \"58aedb13-1685-4dfb-a138-af48cd8e52a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:18.114529 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.114509 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aedb13-1685-4dfb-a138-af48cd8e52a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv\" (UID: \"58aedb13-1685-4dfb-a138-af48cd8e52a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:18.127258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.127233 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf"] Nov 03 22:22:18.215858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.215824 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpxc\" (UniqueName: \"kubernetes.io/projected/2a594be9-4d23-4125-9a34-76e2febb1936-kube-api-access-5cpxc\") pod \"payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf\" (UID: \"2a594be9-4d23-4125-9a34-76e2febb1936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:18.216048 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.215884 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aedb13-1685-4dfb-a138-af48cd8e52a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv\" (UID: \"58aedb13-1685-4dfb-a138-af48cd8e52a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:18.216048 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.215922 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a594be9-4d23-4125-9a34-76e2febb1936-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf\" (UID: \"2a594be9-4d23-4125-9a34-76e2febb1936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:18.216225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.216207 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvkx\" (UniqueName: \"kubernetes.io/projected/58aedb13-1685-4dfb-a138-af48cd8e52a9-kube-api-access-5rvkx\") pod \"payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv\" (UID: \"58aedb13-1685-4dfb-a138-af48cd8e52a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:18.217348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.217320 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aedb13-1685-4dfb-a138-af48cd8e52a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv\" (UID: \"58aedb13-1685-4dfb-a138-af48cd8e52a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:18.238201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.238169 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvkx\" (UniqueName: \"kubernetes.io/projected/58aedb13-1685-4dfb-a138-af48cd8e52a9-kube-api-access-5rvkx\") pod \"payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv\" (UID: \"58aedb13-1685-4dfb-a138-af48cd8e52a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:18.317221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.317135 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a594be9-4d23-4125-9a34-76e2febb1936-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf\" (UID: \"2a594be9-4d23-4125-9a34-76e2febb1936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:18.317221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.317204 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpxc\" (UniqueName: \"kubernetes.io/projected/2a594be9-4d23-4125-9a34-76e2febb1936-kube-api-access-5cpxc\") pod \"payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf\" (UID: \"2a594be9-4d23-4125-9a34-76e2febb1936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:18.317878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.317858 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a594be9-4d23-4125-9a34-76e2febb1936-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf\" (UID: \"2a594be9-4d23-4125-9a34-76e2febb1936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:18.338365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.338342 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpxc\" (UniqueName: \"kubernetes.io/projected/2a594be9-4d23-4125-9a34-76e2febb1936-kube-api-access-5cpxc\") pod \"payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf\" (UID: \"2a594be9-4d23-4125-9a34-76e2febb1936\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:18.359649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.359626 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:18.388432 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:22:18.388393 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:22:18.431261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.428674 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:18.657251 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.657228 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv"] Nov 03 22:22:18.659413 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:18.659382 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aedb13_1685_4dfb_a138_af48cd8e52a9.slice/crio-078cf241e8474c3cf8ad9a85c525edca44ff1b8800691b3cef7108336aac63c7 WatchSource:0}: Error finding container 078cf241e8474c3cf8ad9a85c525edca44ff1b8800691b3cef7108336aac63c7: Status 404 returned error can't find the container with id 078cf241e8474c3cf8ad9a85c525edca44ff1b8800691b3cef7108336aac63c7 Nov 03 22:22:18.731944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:18.731156 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf"] Nov 03 22:22:18.742113 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:18.742087 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a594be9_4d23_4125_9a34_76e2febb1936.slice/crio-f7d1fb476743531b3c9f962a263df5681c148ca2cde8ea6ef378f76b002b9f38 WatchSource:0}: Error finding container f7d1fb476743531b3c9f962a263df5681c148ca2cde8ea6ef378f76b002b9f38: Status 404 returned error can't find the container with id f7d1fb476743531b3c9f962a263df5681c148ca2cde8ea6ef378f76b002b9f38 Nov 03 22:22:19.143406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.143281 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" event={"ID":"58aedb13-1685-4dfb-a138-af48cd8e52a9","Type":"ContainerStarted","Data":"6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9"} Nov 03 22:22:19.143406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.143380 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:19.143917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.143412 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" event={"ID":"58aedb13-1685-4dfb-a138-af48cd8e52a9","Type":"ContainerStarted","Data":"078cf241e8474c3cf8ad9a85c525edca44ff1b8800691b3cef7108336aac63c7"} Nov 03 22:22:19.144922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.144901 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" event={"ID":"2a594be9-4d23-4125-9a34-76e2febb1936","Type":"ContainerStarted","Data":"00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512"} Nov 03 22:22:19.145054 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.144938 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" event={"ID":"2a594be9-4d23-4125-9a34-76e2febb1936","Type":"ContainerStarted","Data":"f7d1fb476743531b3c9f962a263df5681c148ca2cde8ea6ef378f76b002b9f38"} Nov 03 22:22:19.145054 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.144959 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:19.182509 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.182462 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" podStartSLOduration=1.182449915 podStartE2EDuration="1.182449915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:19.18114984 +0000 UTC m=+565.585090815" watchObservedRunningTime="2026-11-03 22:22:19.182449915 +0000 UTC m=+565.586390902" Nov 03 22:22:19.221047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.220989 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" podStartSLOduration=1.220974721 podStartE2EDuration="1.220974721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:19.219582564 +0000 UTC m=+565.623523535" watchObservedRunningTime="2026-11-03 22:22:19.220974721 +0000 UTC m=+565.624915696" Nov 03 22:22:19.606086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.606053 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr"] Nov 03 22:22:19.609792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.609771 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:19.625427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.625401 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-2d619f\"" Nov 03 22:22:19.628127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.628106 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr"] Nov 03 22:22:19.631632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.631608 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-2d619f-dockercfg-wnkf7\"" Nov 03 22:22:19.716423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.715608 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl"] Nov 03 22:22:19.722546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.722512 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:19.731905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.731871 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f387f8d-1889-4269-b3b2-29fe3ba25667-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr\" (UID: \"5f387f8d-1889-4269-b3b2-29fe3ba25667\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:19.732228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.732207 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lksb4\" (UniqueName: \"kubernetes.io/projected/5f387f8d-1889-4269-b3b2-29fe3ba25667-kube-api-access-lksb4\") pod \"payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr\" (UID: \"5f387f8d-1889-4269-b3b2-29fe3ba25667\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:19.737940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.737915 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl"] Nov 03 22:22:19.833082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.833039 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmm5l\" (UniqueName: \"kubernetes.io/projected/76294da7-a114-4504-8530-a7bb4b3136be-kube-api-access-cmm5l\") pod \"payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl\" (UID: \"76294da7-a114-4504-8530-a7bb4b3136be\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:19.833245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.833096 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76294da7-a114-4504-8530-a7bb4b3136be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl\" (UID: \"76294da7-a114-4504-8530-a7bb4b3136be\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:19.833245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.833227 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksb4\" (UniqueName: \"kubernetes.io/projected/5f387f8d-1889-4269-b3b2-29fe3ba25667-kube-api-access-lksb4\") pod \"payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr\" (UID: \"5f387f8d-1889-4269-b3b2-29fe3ba25667\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:19.833348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.833305 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f387f8d-1889-4269-b3b2-29fe3ba25667-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr\" (UID: \"5f387f8d-1889-4269-b3b2-29fe3ba25667\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:19.833970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.833953 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f387f8d-1889-4269-b3b2-29fe3ba25667-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr\" (UID: \"5f387f8d-1889-4269-b3b2-29fe3ba25667\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:19.869468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.869272 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksb4\" (UniqueName: \"kubernetes.io/projected/5f387f8d-1889-4269-b3b2-29fe3ba25667-kube-api-access-lksb4\") pod \"payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr\" (UID: \"5f387f8d-1889-4269-b3b2-29fe3ba25667\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:19.923387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.923353 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:19.938037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.938003 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmm5l\" (UniqueName: \"kubernetes.io/projected/76294da7-a114-4504-8530-a7bb4b3136be-kube-api-access-cmm5l\") pod \"payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl\" (UID: \"76294da7-a114-4504-8530-a7bb4b3136be\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:19.938148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.938070 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76294da7-a114-4504-8530-a7bb4b3136be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl\" (UID: \"76294da7-a114-4504-8530-a7bb4b3136be\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:19.938936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.938913 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76294da7-a114-4504-8530-a7bb4b3136be-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl\" (UID: \"76294da7-a114-4504-8530-a7bb4b3136be\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:19.959498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:19.959472 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmm5l\" (UniqueName: \"kubernetes.io/projected/76294da7-a114-4504-8530-a7bb4b3136be-kube-api-access-cmm5l\") pod \"payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl\" (UID: \"76294da7-a114-4504-8530-a7bb4b3136be\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:20.043138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.043105 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:20.227144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.227112 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr"] Nov 03 22:22:20.230356 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:20.230323 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f387f8d_1889_4269_b3b2_29fe3ba25667.slice/crio-7cb0561ad861b93bde63d6e62693d9a0193252cc6aa369e0a0e93e319584f326 WatchSource:0}: Error finding container 7cb0561ad861b93bde63d6e62693d9a0193252cc6aa369e0a0e93e319584f326: Status 404 returned error can't find the container with id 7cb0561ad861b93bde63d6e62693d9a0193252cc6aa369e0a0e93e319584f326 Nov 03 22:22:20.341547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.341514 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl"] Nov 03 22:22:20.343390 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:20.343356 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76294da7_a114_4504_8530_a7bb4b3136be.slice/crio-94242b1a0704c9feb5bc78272aafc60a3255ad060acf76944546840099599142 WatchSource:0}: Error finding container 94242b1a0704c9feb5bc78272aafc60a3255ad060acf76944546840099599142: Status 404 returned error can't find the container with id 94242b1a0704c9feb5bc78272aafc60a3255ad060acf76944546840099599142 Nov 03 22:22:20.392211 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.392183 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44"] Nov 03 22:22:20.397236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.397215 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:20.414235 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.414211 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e19850\"" Nov 03 22:22:20.415387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.415368 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44"] Nov 03 22:22:20.442713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.442681 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfgd6\" (UniqueName: \"kubernetes.io/projected/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-kube-api-access-zfgd6\") pod \"payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44\" (UID: \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:20.442860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.442792 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44\" (UID: \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:20.543408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.543319 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfgd6\" (UniqueName: \"kubernetes.io/projected/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-kube-api-access-zfgd6\") pod \"payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44\" (UID: \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:20.543408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.543384 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44\" (UID: \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:20.544148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.544126 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44\" (UID: \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:20.564506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.564476 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfgd6\" (UniqueName: \"kubernetes.io/projected/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-kube-api-access-zfgd6\") pod \"payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44\" (UID: \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:20.709885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:20.709846 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:21.025536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.025507 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44"] Nov 03 22:22:21.027703 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:21.027677 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1f1640_d40b_4e0f_ae1f_2e1d53b5a4f7.slice/crio-a5a616f7d09a101591dc56843e22b2256f502a1d154b907df432bcb75b277806 WatchSource:0}: Error finding container a5a616f7d09a101591dc56843e22b2256f502a1d154b907df432bcb75b277806: Status 404 returned error can't find the container with id a5a616f7d09a101591dc56843e22b2256f502a1d154b907df432bcb75b277806 Nov 03 22:22:21.157784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.157738 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" event={"ID":"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7","Type":"ContainerStarted","Data":"56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d"} Nov 03 22:22:21.157941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.157811 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" event={"ID":"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7","Type":"ContainerStarted","Data":"a5a616f7d09a101591dc56843e22b2256f502a1d154b907df432bcb75b277806"} Nov 03 22:22:21.157941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.157843 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:21.159577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.159551 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" event={"ID":"76294da7-a114-4504-8530-a7bb4b3136be","Type":"ContainerStarted","Data":"88034564e11c074b09a9c81730153d0823e3080e3eae8431b19e84ea9f76db8e"} Nov 03 22:22:21.159714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.159598 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" event={"ID":"76294da7-a114-4504-8530-a7bb4b3136be","Type":"ContainerStarted","Data":"94242b1a0704c9feb5bc78272aafc60a3255ad060acf76944546840099599142"} Nov 03 22:22:21.159947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.159721 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:21.161234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.161212 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" event={"ID":"5f387f8d-1889-4269-b3b2-29fe3ba25667","Type":"ContainerStarted","Data":"72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35"} Nov 03 22:22:21.161365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.161247 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" event={"ID":"5f387f8d-1889-4269-b3b2-29fe3ba25667","Type":"ContainerStarted","Data":"7cb0561ad861b93bde63d6e62693d9a0193252cc6aa369e0a0e93e319584f326"} Nov 03 22:22:21.161365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.161266 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:21.202677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.202621 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" podStartSLOduration=1.202603097 podStartE2EDuration="1.202603097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:21.201404763 +0000 UTC m=+567.605345737" watchObservedRunningTime="2026-11-03 22:22:21.202603097 +0000 UTC m=+567.606544074" Nov 03 22:22:21.260947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.260895 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" podStartSLOduration=2.260881411 podStartE2EDuration="2.260881411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:21.258790845 +0000 UTC m=+567.662731820" watchObservedRunningTime="2026-11-03 22:22:21.260881411 +0000 UTC m=+567.664822386" Nov 03 22:22:21.295260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.295163 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" podStartSLOduration=2.295146392 podStartE2EDuration="2.295146392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:21.294772393 +0000 UTC m=+567.698713370" watchObservedRunningTime="2026-11-03 22:22:21.295146392 +0000 UTC m=+567.699087372" Nov 03 22:22:21.355626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.355597 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2"] Nov 03 22:22:21.359378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.359360 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:21.375316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.375275 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-553753\"" Nov 03 22:22:21.375460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.375427 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-553753-dockercfg-tm7gg\"" Nov 03 22:22:21.378039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.378015 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2"] Nov 03 22:22:21.452338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.452047 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qm8\" (UniqueName: \"kubernetes.io/projected/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-kube-api-access-98qm8\") pod \"payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2\" (UID: \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:21.452338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.452178 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2\" (UID: \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:21.553330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.553238 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qm8\" (UniqueName: \"kubernetes.io/projected/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-kube-api-access-98qm8\") pod \"payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2\" (UID: \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:21.553330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.553322 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2\" (UID: \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:21.553992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.553975 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2\" (UID: \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:21.576454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.576425 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qm8\" (UniqueName: \"kubernetes.io/projected/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-kube-api-access-98qm8\") pod \"payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2\" (UID: \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:21.672566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.672536 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:21.978274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:21.978249 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2"] Nov 03 22:22:21.980584 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:21.980559 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ec5663_6735_4dab_b18d_a3f3b722c2a2.slice/crio-30b7f0baf37bbb3a96bfb1d2de70f9a21f870537269acb43f17618395d03440a WatchSource:0}: Error finding container 30b7f0baf37bbb3a96bfb1d2de70f9a21f870537269acb43f17618395d03440a: Status 404 returned error can't find the container with id 30b7f0baf37bbb3a96bfb1d2de70f9a21f870537269acb43f17618395d03440a Nov 03 22:22:22.168321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.168255 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" event={"ID":"a7ec5663-6735-4dab-b18d-a3f3b722c2a2","Type":"ContainerStarted","Data":"eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd"} Nov 03 22:22:22.168501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.168333 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" event={"ID":"a7ec5663-6735-4dab-b18d-a3f3b722c2a2","Type":"ContainerStarted","Data":"30b7f0baf37bbb3a96bfb1d2de70f9a21f870537269acb43f17618395d03440a"} Nov 03 22:22:22.168770 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.168750 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:22.207901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.207847 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" podStartSLOduration=1.207832534 podStartE2EDuration="1.207832534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:22.20736534 +0000 UTC m=+568.611306315" watchObservedRunningTime="2026-11-03 22:22:22.207832534 +0000 UTC m=+568.611773508" Nov 03 22:22:22.787733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.787699 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp"] Nov 03 22:22:22.791626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.791603 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:22.811422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.811385 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-434ce3\"" Nov 03 22:22:22.821526 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.821495 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-434ce3-dockercfg-t4h4q\"" Nov 03 22:22:22.823367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.823344 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp"] Nov 03 22:22:22.859033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.859002 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw"] Nov 03 22:22:22.863197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.863176 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:22.867588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.867563 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41e7d90-969b-4616-be62-c357c98e70a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp\" (UID: \"f41e7d90-969b-4616-be62-c357c98e70a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:22.867697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.867640 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngf8\" (UniqueName: \"kubernetes.io/projected/f41e7d90-969b-4616-be62-c357c98e70a0-kube-api-access-hngf8\") pod \"payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp\" (UID: \"f41e7d90-969b-4616-be62-c357c98e70a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:22.880863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.880834 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw"] Nov 03 22:22:22.969010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.968975 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a2f30c-1a18-40be-b501-0571787facdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw\" (UID: \"c4a2f30c-1a18-40be-b501-0571787facdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:22.969182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.969028 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41e7d90-969b-4616-be62-c357c98e70a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp\" (UID: \"f41e7d90-969b-4616-be62-c357c98e70a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:22.969182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.969151 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfrh4\" (UniqueName: \"kubernetes.io/projected/c4a2f30c-1a18-40be-b501-0571787facdc-kube-api-access-vfrh4\") pod \"payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw\" (UID: \"c4a2f30c-1a18-40be-b501-0571787facdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:22.969248 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.969182 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngf8\" (UniqueName: \"kubernetes.io/projected/f41e7d90-969b-4616-be62-c357c98e70a0-kube-api-access-hngf8\") pod \"payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp\" (UID: \"f41e7d90-969b-4616-be62-c357c98e70a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:22.969788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.969770 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41e7d90-969b-4616-be62-c357c98e70a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp\" (UID: \"f41e7d90-969b-4616-be62-c357c98e70a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:22.990410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:22.990381 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngf8\" (UniqueName: \"kubernetes.io/projected/f41e7d90-969b-4616-be62-c357c98e70a0-kube-api-access-hngf8\") pod \"payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp\" (UID: \"f41e7d90-969b-4616-be62-c357c98e70a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:23.070193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:23.070111 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfrh4\" (UniqueName: \"kubernetes.io/projected/c4a2f30c-1a18-40be-b501-0571787facdc-kube-api-access-vfrh4\") pod \"payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw\" (UID: \"c4a2f30c-1a18-40be-b501-0571787facdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:23.070356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:23.070224 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a2f30c-1a18-40be-b501-0571787facdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw\" (UID: \"c4a2f30c-1a18-40be-b501-0571787facdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:23.070940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:23.070920 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a2f30c-1a18-40be-b501-0571787facdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw\" (UID: \"c4a2f30c-1a18-40be-b501-0571787facdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:23.093876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:23.093849 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfrh4\" (UniqueName: \"kubernetes.io/projected/c4a2f30c-1a18-40be-b501-0571787facdc-kube-api-access-vfrh4\") pod \"payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw\" (UID: \"c4a2f30c-1a18-40be-b501-0571787facdc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:23.107775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:23.107747 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:23.176413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:23.176098 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:23.307825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:23.307777 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:22:23.408914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:23.408888 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp"] Nov 03 22:22:23.411571 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:23.411541 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41e7d90_969b_4616_be62_c357c98e70a0.slice/crio-87bea4839395af79c58c6a00413c6e8dcc01b8e0d190190e9f946adc87f4b2e1 WatchSource:0}: Error finding container 87bea4839395af79c58c6a00413c6e8dcc01b8e0d190190e9f946adc87f4b2e1: Status 404 returned error can't find the container with id 87bea4839395af79c58c6a00413c6e8dcc01b8e0d190190e9f946adc87f4b2e1 Nov 03 22:22:23.480129 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:23.480106 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw"] Nov 03 22:22:23.483099 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:23.483075 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a2f30c_1a18_40be_b501_0571787facdc.slice/crio-7b99bc112bc457f2e076f47f974c270c022c959f5504ad095dc99651c410ea82 WatchSource:0}: Error finding container 7b99bc112bc457f2e076f47f974c270c022c959f5504ad095dc99651c410ea82: Status 404 returned error can't find the container with id 7b99bc112bc457f2e076f47f974c270c022c959f5504ad095dc99651c410ea82 Nov 03 22:22:24.182076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.182028 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" event={"ID":"c4a2f30c-1a18-40be-b501-0571787facdc","Type":"ContainerStarted","Data":"fdd152ad35bfd985058efb2816fe9ac1477d3ea9ccca1feb901001a864df37dc"} Nov 03 22:22:24.182562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.182088 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" event={"ID":"c4a2f30c-1a18-40be-b501-0571787facdc","Type":"ContainerStarted","Data":"7b99bc112bc457f2e076f47f974c270c022c959f5504ad095dc99651c410ea82"} Nov 03 22:22:24.182562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.182165 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:24.183704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.183674 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" event={"ID":"f41e7d90-969b-4616-be62-c357c98e70a0","Type":"ContainerStarted","Data":"6512f0cff1d703cca3fad666471bddec4f64daa8d001438a211196959000f37a"} Nov 03 22:22:24.183810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.183719 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" event={"ID":"f41e7d90-969b-4616-be62-c357c98e70a0","Type":"ContainerStarted","Data":"87bea4839395af79c58c6a00413c6e8dcc01b8e0d190190e9f946adc87f4b2e1"} Nov 03 22:22:24.183864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.183855 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:24.219062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.219019 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" podStartSLOduration=2.219005752 podStartE2EDuration="2.219005752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:24.217137473 +0000 UTC m=+570.621078447" watchObservedRunningTime="2026-11-03 22:22:24.219005752 +0000 UTC m=+570.622946736" Nov 03 22:22:24.255244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.255198 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" podStartSLOduration=2.255184947 podStartE2EDuration="2.255184947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:24.253037654 +0000 UTC m=+570.656978694" watchObservedRunningTime="2026-11-03 22:22:24.255184947 +0000 UTC m=+570.659125921" Nov 03 22:22:24.487367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.487274 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh"] Nov 03 22:22:24.491188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.491170 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:24.506993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.506964 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5187e1\"" Nov 03 22:22:24.507187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.507164 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5187e1-dockercfg-xtf9q\"" Nov 03 22:22:24.510595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.510186 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh"] Nov 03 22:22:24.594319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.593786 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml"] Nov 03 22:22:24.596343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.595458 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2aab3-c989-42f3-af29-088f1a463ae0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh\" (UID: \"46d2aab3-c989-42f3-af29-088f1a463ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:24.596343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.595510 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28j27\" (UniqueName: \"kubernetes.io/projected/46d2aab3-c989-42f3-af29-088f1a463ae0-kube-api-access-28j27\") pod \"payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh\" (UID: \"46d2aab3-c989-42f3-af29-088f1a463ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:24.599112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.599092 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:24.612854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.612823 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml"] Nov 03 22:22:24.696350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.696316 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p54mr\" (UniqueName: \"kubernetes.io/projected/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-kube-api-access-p54mr\") pod \"payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml\" (UID: \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:24.696603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.696374 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml\" (UID: \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:24.696603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.696496 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2aab3-c989-42f3-af29-088f1a463ae0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh\" (UID: \"46d2aab3-c989-42f3-af29-088f1a463ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:24.696603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.696534 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28j27\" (UniqueName: \"kubernetes.io/projected/46d2aab3-c989-42f3-af29-088f1a463ae0-kube-api-access-28j27\") pod \"payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh\" (UID: \"46d2aab3-c989-42f3-af29-088f1a463ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:24.697205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.697186 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2aab3-c989-42f3-af29-088f1a463ae0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh\" (UID: \"46d2aab3-c989-42f3-af29-088f1a463ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:24.717768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.717743 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28j27\" (UniqueName: \"kubernetes.io/projected/46d2aab3-c989-42f3-af29-088f1a463ae0-kube-api-access-28j27\") pod \"payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh\" (UID: \"46d2aab3-c989-42f3-af29-088f1a463ae0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:24.798340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.798222 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p54mr\" (UniqueName: \"kubernetes.io/projected/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-kube-api-access-p54mr\") pod \"payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml\" (UID: \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:24.798340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.798268 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml\" (UID: \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:24.799014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.798991 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml\" (UID: \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:24.811939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.811911 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:24.819161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.819132 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p54mr\" (UniqueName: \"kubernetes.io/projected/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-kube-api-access-p54mr\") pod \"payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml\" (UID: \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:24.912052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:24.912022 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:25.133713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.133681 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh"] Nov 03 22:22:25.136258 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:25.136230 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d2aab3_c989_42f3_af29_088f1a463ae0.slice/crio-7ac9fd7dec7508f73c840ede3da910e509b4d2e6756d7715294b508f06e308ff WatchSource:0}: Error finding container 7ac9fd7dec7508f73c840ede3da910e509b4d2e6756d7715294b508f06e308ff: Status 404 returned error can't find the container with id 7ac9fd7dec7508f73c840ede3da910e509b4d2e6756d7715294b508f06e308ff Nov 03 22:22:25.189881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.189846 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" event={"ID":"46d2aab3-c989-42f3-af29-088f1a463ae0","Type":"ContainerStarted","Data":"7ac9fd7dec7508f73c840ede3da910e509b4d2e6756d7715294b508f06e308ff"} Nov 03 22:22:25.265319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.242005 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml"] Nov 03 22:22:25.269751 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:25.269710 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf4c6af_f700_4a03_8491_a5e51fd4dab2.slice/crio-c48056e9048afbd5f36d12630d161ae5bea3a13aaaacec11bb6756bb6c1ff2df WatchSource:0}: Error finding container c48056e9048afbd5f36d12630d161ae5bea3a13aaaacec11bb6756bb6c1ff2df: Status 404 returned error can't find the container with id c48056e9048afbd5f36d12630d161ae5bea3a13aaaacec11bb6756bb6c1ff2df Nov 03 22:22:25.362352 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.361345 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn"] Nov 03 22:22:25.367105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.366170 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:25.376391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.376363 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn"] Nov 03 22:22:25.379121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.379098 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-88a4d9\"" Nov 03 22:22:25.383793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.383749 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-88a4d9-dockercfg-9wmxp\"" Nov 03 22:22:25.404308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.404270 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f255b355-f63f-494e-96df-40136203d990-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn\" (UID: \"f255b355-f63f-494e-96df-40136203d990\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:25.404441 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.404386 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8l5b\" (UniqueName: \"kubernetes.io/projected/f255b355-f63f-494e-96df-40136203d990-kube-api-access-l8l5b\") pod \"payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn\" (UID: \"f255b355-f63f-494e-96df-40136203d990\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:25.505838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.505804 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f255b355-f63f-494e-96df-40136203d990-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn\" (UID: \"f255b355-f63f-494e-96df-40136203d990\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:25.506020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.505851 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8l5b\" (UniqueName: \"kubernetes.io/projected/f255b355-f63f-494e-96df-40136203d990-kube-api-access-l8l5b\") pod \"payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn\" (UID: \"f255b355-f63f-494e-96df-40136203d990\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:25.506536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.506517 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f255b355-f63f-494e-96df-40136203d990-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn\" (UID: \"f255b355-f63f-494e-96df-40136203d990\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:25.527012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.526986 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8l5b\" (UniqueName: \"kubernetes.io/projected/f255b355-f63f-494e-96df-40136203d990-kube-api-access-l8l5b\") pod \"payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn\" (UID: \"f255b355-f63f-494e-96df-40136203d990\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:25.685223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:25.685186 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:26.002739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.002715 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn"] Nov 03 22:22:26.004340 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:26.004315 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf255b355_f63f_494e_96df_40136203d990.slice/crio-56ed9440e22fe183203b18e90a505fa48bce8a6faa1ded3a0c76efee3fff5d17 WatchSource:0}: Error finding container 56ed9440e22fe183203b18e90a505fa48bce8a6faa1ded3a0c76efee3fff5d17: Status 404 returned error can't find the container with id 56ed9440e22fe183203b18e90a505fa48bce8a6faa1ded3a0c76efee3fff5d17 Nov 03 22:22:26.197073 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.197041 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" event={"ID":"46d2aab3-c989-42f3-af29-088f1a463ae0","Type":"ContainerStarted","Data":"c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50"} Nov 03 22:22:26.197809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.197793 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:26.198977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.198957 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" event={"ID":"f255b355-f63f-494e-96df-40136203d990","Type":"ContainerStarted","Data":"6786fb16605e0d6f78d83480e6445d8da693e7f7c215703a1334cf29de55480e"} Nov 03 22:22:26.199024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.198997 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" event={"ID":"f255b355-f63f-494e-96df-40136203d990","Type":"ContainerStarted","Data":"56ed9440e22fe183203b18e90a505fa48bce8a6faa1ded3a0c76efee3fff5d17"} Nov 03 22:22:26.199446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.199432 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:26.200666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.200651 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" event={"ID":"bdf4c6af-f700-4a03-8491-a5e51fd4dab2","Type":"ContainerStarted","Data":"f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250"} Nov 03 22:22:26.200714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.200680 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" event={"ID":"bdf4c6af-f700-4a03-8491-a5e51fd4dab2","Type":"ContainerStarted","Data":"c48056e9048afbd5f36d12630d161ae5bea3a13aaaacec11bb6756bb6c1ff2df"} Nov 03 22:22:26.201040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.201026 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:26.235039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.234984 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" podStartSLOduration=2.234970096 podStartE2EDuration="2.234970096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:26.234077939 +0000 UTC m=+572.638018913" watchObservedRunningTime="2026-11-03 22:22:26.234970096 +0000 UTC m=+572.638911072" Nov 03 22:22:26.285432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.285237 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" podStartSLOduration=2.285221295 podStartE2EDuration="2.285221295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:26.284565509 +0000 UTC m=+572.688506486" watchObservedRunningTime="2026-11-03 22:22:26.285221295 +0000 UTC m=+572.689162271" Nov 03 22:22:26.320692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:26.320634 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" podStartSLOduration=1.320617521 podStartE2EDuration="1.320617521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:26.319758704 +0000 UTC m=+572.723699691" watchObservedRunningTime="2026-11-03 22:22:26.320617521 +0000 UTC m=+572.724558497" Nov 03 22:22:28.088155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:28.088117 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:22:30.166659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:30.166394 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 22:22:30.167130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:30.166810 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 22:22:31.166780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:31.166431 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:22:31.166780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:31.166497 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:22:32.171618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:32.171587 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 22:22:32.172054 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:32.171668 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 22:22:32.172054 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:32.171817 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 22:22:34.188311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:34.188259 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 22:22:35.193246 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:35.193218 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 22:22:35.193703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:35.193442 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 22:22:35.294320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:35.293586 4903 generic.go:356] "Generic (PLEG): container finished" podID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerID="70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955" exitCode=0 Nov 03 22:22:35.294320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:35.293729 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" event={"ID":"acf9ce64-ce17-4f22-9a81-b038a7987369","Type":"ContainerDied","Data":"70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955"} Nov 03 22:22:35.294320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:35.294164 4903 scope.go:122] "RemoveContainer" containerID="70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955" Nov 03 22:22:35.297041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:35.296999 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:35.297175 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:35.297097 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:35.297175 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:35.297144 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:36.300993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:36.300915 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" event={"ID":"acf9ce64-ce17-4f22-9a81-b038a7987369","Type":"ContainerStarted","Data":"262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a"} Nov 03 22:22:36.671568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:36.671531 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:36.671783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:36.671762 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:22:36.671852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:36.671811 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:22:37.666760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:37.666719 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:22:37.667147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:37.666804 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:22:38.166324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.165639 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs"] Nov 03 22:22:38.173194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.173069 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.197161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.197128 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs"] Nov 03 22:22:38.199087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.199062 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"kube-root-ca.crt\"" Nov 03 22:22:38.199087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.199077 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"openshift-service-ca.crt\"" Nov 03 22:22:38.215936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.215908 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 22:22:38.216111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.216006 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 22:22:38.216340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.216320 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 22:22:38.270971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.270944 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"default-dockercfg-krbfz\"" Nov 03 22:22:38.271471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.271446 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"simulator-w6-2d619f-kserve-self-signed-certs\"" Nov 03 22:22:38.308749 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.308717 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-dshm\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.308927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.308786 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-model-cache\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.308927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.308849 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tmp-dir\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.309110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.308935 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-home\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.309110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.309043 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tls-certs\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.309110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.309079 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvrd\" (UniqueName: \"kubernetes.io/projected/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-kube-api-access-8jvrd\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.411733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.411701 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-model-cache\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.411958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.411778 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tmp-dir\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.411958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.411864 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-home\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.411958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.411919 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tls-certs\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.411958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.411952 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvrd\" (UniqueName: \"kubernetes.io/projected/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-kube-api-access-8jvrd\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.412186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.412002 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-dshm\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.412186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.412163 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-model-cache\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.412319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.412198 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tmp-dir\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.412662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.412433 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-home\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.414381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.414358 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-dshm\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.414938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.414918 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tls-certs\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.434906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.434847 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvrd\" (UniqueName: \"kubernetes.io/projected/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-kube-api-access-8jvrd\") pod \"simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.494732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.494690 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:38.669327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.664687 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:22:38.669327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.664744 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:22:38.822236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:38.821545 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs"] Nov 03 22:22:39.321806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.321728 4903 generic.go:356] "Generic (PLEG): container finished" podID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerID="9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5" exitCode=0 Nov 03 22:22:39.322039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.321799 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" event={"ID":"849e7c9f-8596-41a4-b9ed-e7472cecadbb","Type":"ContainerDied","Data":"9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5"} Nov 03 22:22:39.322210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.322192 4903 scope.go:122] "RemoveContainer" containerID="9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5" Nov 03 22:22:39.323508 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.323479 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" event={"ID":"806bbbeb-b428-44e2-95c1-4cf3282eb1cd","Type":"ContainerStarted","Data":"f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3"} Nov 03 22:22:39.323594 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.323498 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:39.323594 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.323570 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:39.323594 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.323510 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" event={"ID":"806bbbeb-b428-44e2-95c1-4cf3282eb1cd","Type":"ContainerStarted","Data":"979e7e2179fe59545b5bd078b8a298fb1d8bd080f01898a25372b0b908b4f47b"} Nov 03 22:22:39.323718 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.323600 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:22:39.398556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.398412 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podStartSLOduration=1.39828609 podStartE2EDuration="1.39828609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:39.397995805 +0000 UTC m=+585.801936783" watchObservedRunningTime="2026-11-03 22:22:39.39828609 +0000 UTC m=+585.802227065" Nov 03 22:22:39.495909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.495857 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:39.667319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.666804 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:22:39.667319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:39.666865 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:22:40.330649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:40.330607 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" event={"ID":"849e7c9f-8596-41a4-b9ed-e7472cecadbb","Type":"ContainerStarted","Data":"21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6"} Nov 03 22:22:40.676316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:40.665730 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 22:22:40.676316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:40.665806 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 22:22:40.795836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:40.795494 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:40.796727 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:40.796699 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 22:22:40.796875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:40.796742 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 22:22:41.709428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:41.709399 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:41.709939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:41.709874 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:41.710614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:41.710593 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:22:41.794521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:41.794491 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 22:22:41.794728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:41.794537 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 22:22:41.875047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:41.874993 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:22:42.795026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:42.794995 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 22:22:42.795398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:42.795059 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 22:22:43.794910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:43.794876 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 22:22:43.795087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:43.794924 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 22:22:44.795283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:44.795250 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 22:22:44.795781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:44.795327 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 22:22:45.119147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:45.119065 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:45.799485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:45.799454 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:45.800552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:45.800531 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:22:48.386810 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:22:48.386770 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:22:48.495893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:48.495860 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:48.508228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:48.508200 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:48.941376 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:48.941344 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm"] Nov 03 22:22:48.944964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:48.944942 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:48.958192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:48.958173 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"kube-root-ca.crt\"" Nov 03 22:22:48.967365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:48.967341 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"simulator-w1-e19850-kserve-self-signed-certs\"" Nov 03 22:22:48.967365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:48.967335 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"openshift-service-ca.crt\"" Nov 03 22:22:48.967526 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:48.967383 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"default-dockercfg-95q47\"" Nov 03 22:22:48.969609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:48.969586 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm"] Nov 03 22:22:49.053984 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.053951 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-dshm\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.053984 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.053984 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-model-cache\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.054196 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.054111 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dc74e3-303e-4920-9d9f-4e417621cb7a-tls-certs\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.054196 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.054181 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-tmp-dir\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.054264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.054211 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-home\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.054264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.054248 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2cs\" (UniqueName: \"kubernetes.io/projected/85dc74e3-303e-4920-9d9f-4e417621cb7a-kube-api-access-2q2cs\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.155470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.155435 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-dshm\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.155470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.155479 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-model-cache\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.155725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.155518 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dc74e3-303e-4920-9d9f-4e417621cb7a-tls-certs\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.155725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.155547 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-tmp-dir\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.155725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.155571 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-home\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.155725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.155612 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2cs\" (UniqueName: \"kubernetes.io/projected/85dc74e3-303e-4920-9d9f-4e417621cb7a-kube-api-access-2q2cs\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.156019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.155998 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-model-cache\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.156102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.156044 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-tmp-dir\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.156158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.156136 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-home\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.158399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.158377 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-dshm\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.158569 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.158552 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dc74e3-303e-4920-9d9f-4e417621cb7a-tls-certs\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.177259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.177229 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2cs\" (UniqueName: \"kubernetes.io/projected/85dc74e3-303e-4920-9d9f-4e417621cb7a-kube-api-access-2q2cs\") pod \"simulator-w1-e19850-kserve-7c89799dcd-rnzhm\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.255522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.255441 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:49.415403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.415375 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:22:49.554949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:49.554923 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm"] Nov 03 22:22:49.557140 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:49.557114 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85dc74e3_303e_4920_9d9f_4e417621cb7a.slice/crio-d12f02c8bca84e954d2d3123e9d2a8d6b2b0b54054e80775599fa993d2919591 WatchSource:0}: Error finding container d12f02c8bca84e954d2d3123e9d2a8d6b2b0b54054e80775599fa993d2919591: Status 404 returned error can't find the container with id d12f02c8bca84e954d2d3123e9d2a8d6b2b0b54054e80775599fa993d2919591 Nov 03 22:22:50.412564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:50.412528 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" event={"ID":"85dc74e3-303e-4920-9d9f-4e417621cb7a","Type":"ContainerStarted","Data":"5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d"} Nov 03 22:22:50.412764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:50.412573 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" event={"ID":"85dc74e3-303e-4920-9d9f-4e417621cb7a","Type":"ContainerStarted","Data":"d12f02c8bca84e954d2d3123e9d2a8d6b2b0b54054e80775599fa993d2919591"} Nov 03 22:22:50.450269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:50.450216 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podStartSLOduration=2.450202323 podStartE2EDuration="2.450202323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:50.448325513 +0000 UTC m=+596.852266479" watchObservedRunningTime="2026-11-03 22:22:50.450202323 +0000 UTC m=+596.854143275" Nov 03 22:22:51.255751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:51.255719 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:52.516655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.516624 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4"] Nov 03 22:22:52.520404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.520377 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.547619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.547590 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4"] Nov 03 22:22:52.554423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.554396 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"default-dockercfg-qg8d5\"" Nov 03 22:22:52.554423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.554420 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"kube-root-ca.crt\"" Nov 03 22:22:52.554610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.554396 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"openshift-service-ca.crt\"" Nov 03 22:22:52.554610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.554477 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"simulator-w5-434ce3-kserve-self-signed-certs\"" Nov 03 22:22:52.590445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.590410 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-model-cache\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.590445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.590443 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzglh\" (UniqueName: \"kubernetes.io/projected/f8c71259-4646-42b5-a78d-ae727da42f34-kube-api-access-nzglh\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.590660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.590528 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-tmp-dir\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.590660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.590589 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-dshm\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.590660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.590629 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c71259-4646-42b5-a78d-ae727da42f34-tls-certs\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.590817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.590665 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-home\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.691240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.691199 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c71259-4646-42b5-a78d-ae727da42f34-tls-certs\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.691460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.691250 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-home\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.691460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.691323 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-model-cache\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.691460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.691345 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzglh\" (UniqueName: \"kubernetes.io/projected/f8c71259-4646-42b5-a78d-ae727da42f34-kube-api-access-nzglh\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.691460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.691396 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-tmp-dir\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.691694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.691461 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-dshm\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.691847 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.691823 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-model-cache\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.691847 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.691837 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-tmp-dir\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.691957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.691879 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-home\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.693623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.693600 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-dshm\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.693938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.693917 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c71259-4646-42b5-a78d-ae727da42f34-tls-certs\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.714683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.714651 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzglh\" (UniqueName: \"kubernetes.io/projected/f8c71259-4646-42b5-a78d-ae727da42f34-kube-api-access-nzglh\") pod \"simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:52.844537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:52.844451 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:53.175566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.175539 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4"] Nov 03 22:22:53.176343 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:22:53.176320 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8c71259_4646_42b5_a78d_ae727da42f34.slice/crio-25ab60798afdedd84038818b259a2a6a8aece89d8fc16d3b4cd433cb324a3916 WatchSource:0}: Error finding container 25ab60798afdedd84038818b259a2a6a8aece89d8fc16d3b4cd433cb324a3916: Status 404 returned error can't find the container with id 25ab60798afdedd84038818b259a2a6a8aece89d8fc16d3b4cd433cb324a3916 Nov 03 22:22:53.431112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.431032 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" event={"ID":"f8c71259-4646-42b5-a78d-ae727da42f34","Type":"ContainerStarted","Data":"3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46"} Nov 03 22:22:53.431112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.431073 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" event={"ID":"f8c71259-4646-42b5-a78d-ae727da42f34","Type":"ContainerStarted","Data":"25ab60798afdedd84038818b259a2a6a8aece89d8fc16d3b4cd433cb324a3916"} Nov 03 22:22:53.473620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.473577 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podStartSLOduration=1.473563853 podStartE2EDuration="1.473563853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:53.470438616 +0000 UTC m=+599.874379603" watchObservedRunningTime="2026-11-03 22:22:53.473563853 +0000 UTC m=+599.877504828" Nov 03 22:22:53.784551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.784469 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:22:53.785716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.785695 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:22:53.792049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.792026 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:22:53.793189 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.793172 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:22:53.844622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.844592 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:22:53.953408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.953376 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5"] Nov 03 22:22:53.956904 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.956881 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:53.971225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.971205 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"openshift-service-ca.crt\"" Nov 03 22:22:53.971447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.971429 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"kube-root-ca.crt\"" Nov 03 22:22:53.980460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.980440 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"simulator-w3-5187e1-kserve-self-signed-certs\"" Nov 03 22:22:53.980870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.980841 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"default-dockercfg-5nm2d\"" Nov 03 22:22:53.983996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:53.983975 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5"] Nov 03 22:22:54.007000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.006970 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvn8d\" (UniqueName: \"kubernetes.io/projected/2a063f54-ad04-4e48-8b3c-246b782ce808-kube-api-access-mvn8d\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.007153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.007031 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-model-cache\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.007153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.007092 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-dshm\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.007233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.007148 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-home\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.007233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.007196 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a063f54-ad04-4e48-8b3c-246b782ce808-tls-certs\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.007313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.007252 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-tmp-dir\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.108206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.108124 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-dshm\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.108206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.108162 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-home\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.108206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.108189 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a063f54-ad04-4e48-8b3c-246b782ce808-tls-certs\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.108524 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.108247 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-tmp-dir\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.108524 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.108323 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvn8d\" (UniqueName: \"kubernetes.io/projected/2a063f54-ad04-4e48-8b3c-246b782ce808-kube-api-access-mvn8d\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.108524 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.108372 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-model-cache\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.108713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.108667 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-home\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.108823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.108690 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-tmp-dir\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.108823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.108745 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-model-cache\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.110403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.110375 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-dshm\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.110726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.110709 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a063f54-ad04-4e48-8b3c-246b782ce808-tls-certs\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.129174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.129150 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvn8d\" (UniqueName: \"kubernetes.io/projected/2a063f54-ad04-4e48-8b3c-246b782ce808-kube-api-access-mvn8d\") pod \"simulator-w3-5187e1-kserve-657d5cc745-57hv5\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.269004 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.268964 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:54.575348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:54.571596 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5"] Nov 03 22:22:55.444503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:55.444468 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" event={"ID":"2a063f54-ad04-4e48-8b3c-246b782ce808","Type":"ContainerStarted","Data":"b930e49abd7fdce7dd4747145e4f49735fe81a03bf0f98d5fc30f37ea5f28044"} Nov 03 22:22:55.444503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:55.444510 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" event={"ID":"2a063f54-ad04-4e48-8b3c-246b782ce808","Type":"ContainerStarted","Data":"52b374986bd65f9e85ca088bb8d93e6be9f3fedae293a874491a627927450d93"} Nov 03 22:22:55.482907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:55.482852 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podStartSLOduration=2.482838514 podStartE2EDuration="2.482838514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:22:55.481448052 +0000 UTC m=+601.885389040" watchObservedRunningTime="2026-11-03 22:22:55.482838514 +0000 UTC m=+601.886779488" Nov 03 22:22:56.269232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:56.269190 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:22:56.854953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:56.854914 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:22:59.256432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:59.256395 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:59.269909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:59.269878 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:22:59.486865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:22:59.486836 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:23:02.845032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:02.844998 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:23:02.857886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:02.857859 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:23:03.505514 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:03.505483 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:23:04.270187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.270152 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:23:04.283001 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.282969 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:23:04.511519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.511483 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:23:04.795360 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.795326 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g"] Nov 03 22:23:04.799951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.799928 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.816431 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.816407 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-dshm\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.816546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.816442 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25fgj\" (UniqueName: \"kubernetes.io/projected/cfc71b2b-0750-4775-ac68-9ddf63d67773-kube-api-access-25fgj\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.816546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.816463 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc71b2b-0750-4775-ac68-9ddf63d67773-tls-certs\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.816546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.816505 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-tmp-dir\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.816783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.816586 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-model-cache\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.816783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.816688 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-home\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.817987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.817965 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"kube-root-ca.crt\"" Nov 03 22:23:04.819749 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.819728 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g"] Nov 03 22:23:04.823561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.823545 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"openshift-service-ca.crt\"" Nov 03 22:23:04.854570 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.854538 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"default-dockercfg-rndwk\"" Nov 03 22:23:04.854702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.854579 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-50d738\"/\"simulator-w4-50d738-kserve-self-signed-certs\"" Nov 03 22:23:04.917770 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.917743 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-home\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.917931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.917826 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-dshm\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.917931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.917854 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25fgj\" (UniqueName: \"kubernetes.io/projected/cfc71b2b-0750-4775-ac68-9ddf63d67773-kube-api-access-25fgj\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.917931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.917874 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc71b2b-0750-4775-ac68-9ddf63d67773-tls-certs\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.917931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.917893 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-tmp-dir\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.918133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.917967 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-model-cache\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.918226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.918196 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-home\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.918373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.918355 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-tmp-dir\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.918373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.918365 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-model-cache\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.920142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.920117 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-dshm\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.920421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.920406 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc71b2b-0750-4775-ac68-9ddf63d67773-tls-certs\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:04.941057 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:04.941035 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25fgj\" (UniqueName: \"kubernetes.io/projected/cfc71b2b-0750-4775-ac68-9ddf63d67773-kube-api-access-25fgj\") pod \"simulator-w4-50d738-kserve-576c765958-mvn6g\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:05.114662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:05.114576 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:05.411978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:05.411949 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g"] Nov 03 22:23:05.413871 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:23:05.413830 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfc71b2b_0750_4775_ac68_9ddf63d67773.slice/crio-e882b3e883ca0ddbb9c59a0168a1835314be333b8b9c849ce0456be890666fa5 WatchSource:0}: Error finding container e882b3e883ca0ddbb9c59a0168a1835314be333b8b9c849ce0456be890666fa5: Status 404 returned error can't find the container with id e882b3e883ca0ddbb9c59a0168a1835314be333b8b9c849ce0456be890666fa5 Nov 03 22:23:05.507439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:05.507399 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" event={"ID":"cfc71b2b-0750-4775-ac68-9ddf63d67773","Type":"ContainerStarted","Data":"6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a"} Nov 03 22:23:05.507624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:05.507452 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" event={"ID":"cfc71b2b-0750-4775-ac68-9ddf63d67773","Type":"ContainerStarted","Data":"e882b3e883ca0ddbb9c59a0168a1835314be333b8b9c849ce0456be890666fa5"} Nov 03 22:23:05.545547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:05.545496 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podStartSLOduration=1.545483632 podStartE2EDuration="1.545483632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:05.543442037 +0000 UTC m=+611.947382994" watchObservedRunningTime="2026-11-03 22:23:05.545483632 +0000 UTC m=+611.949424607" Nov 03 22:23:06.117074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:06.116369 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:12.788036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:12.788001 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:23:15.115392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:15.115348 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:15.130449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:15.130422 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:15.652901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:15.652871 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:23:18.383193 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:23:18.383160 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:23:20.793943 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.793909 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb"] Nov 03 22:23:20.798188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.798158 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:20.827025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.826993 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2d619f\"/\"premium-w6-2d619f-kserve-self-signed-certs\"" Nov 03 22:23:20.829617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.829588 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb"] Nov 03 22:23:20.905074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.905043 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-dshm\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:20.905246 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.905088 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-home\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:20.905246 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.905171 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-model-cache\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:20.905350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.905259 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfgp\" (UniqueName: \"kubernetes.io/projected/599f7b09-4f28-499c-bb80-cf9811e68a73-kube-api-access-xdfgp\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:20.905350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.905286 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599f7b09-4f28-499c-bb80-cf9811e68a73-tls-certs\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:20.905461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:20.905445 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-tmp-dir\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.006191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.006161 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-tmp-dir\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.006361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.006215 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-dshm\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.006361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.006257 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-home\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.006361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.006283 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-model-cache\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.006533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.006361 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfgp\" (UniqueName: \"kubernetes.io/projected/599f7b09-4f28-499c-bb80-cf9811e68a73-kube-api-access-xdfgp\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.006533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.006395 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599f7b09-4f28-499c-bb80-cf9811e68a73-tls-certs\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.006670 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.006644 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-tmp-dir\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.006731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.006694 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-home\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.006796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.006777 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-model-cache\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.008663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.008642 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-dshm\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.008878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.008864 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599f7b09-4f28-499c-bb80-cf9811e68a73-tls-certs\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.032505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.032474 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfgp\" (UniqueName: \"kubernetes.io/projected/599f7b09-4f28-499c-bb80-cf9811e68a73-kube-api-access-xdfgp\") pod \"premium-w6-2d619f-kserve-75c689cbb8-zlsbb\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.112590 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.112508 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:21.415639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.415615 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb"] Nov 03 22:23:21.417321 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:23:21.417276 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod599f7b09_4f28_499c_bb80_cf9811e68a73.slice/crio-a73ebd7e5d0f732c34ab145628ac7fd0854db82a175a860a1d23c86d1a14d241 WatchSource:0}: Error finding container a73ebd7e5d0f732c34ab145628ac7fd0854db82a175a860a1d23c86d1a14d241: Status 404 returned error can't find the container with id a73ebd7e5d0f732c34ab145628ac7fd0854db82a175a860a1d23c86d1a14d241 Nov 03 22:23:21.676889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.676795 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" event={"ID":"599f7b09-4f28-499c-bb80-cf9811e68a73","Type":"ContainerStarted","Data":"8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2"} Nov 03 22:23:21.676889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.676845 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" event={"ID":"599f7b09-4f28-499c-bb80-cf9811e68a73","Type":"ContainerStarted","Data":"a73ebd7e5d0f732c34ab145628ac7fd0854db82a175a860a1d23c86d1a14d241"} Nov 03 22:23:21.716156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:21.716106 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podStartSLOduration=1.716091968 podStartE2EDuration="1.716091968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:21.713515947 +0000 UTC m=+628.117456922" watchObservedRunningTime="2026-11-03 22:23:21.716091968 +0000 UTC m=+628.120032942" Nov 03 22:23:22.113418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:22.113339 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:27.154364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:27.154311 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:23:31.112972 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:31.112934 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:31.125665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:31.125638 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:31.749071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:31.749039 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:23:39.779788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.779754 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk"] Nov 03 22:23:39.783729 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.783704 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.802989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.802959 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-tmp-dir\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.802989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.802993 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2b69ca-544e-4521-97a2-09a6cecf868a-tls-certs\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.803210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.803025 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-home\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.803210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.803066 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-dshm\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.803210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.803093 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p742\" (UniqueName: \"kubernetes.io/projected/2d2b69ca-544e-4521-97a2-09a6cecf868a-kube-api-access-4p742\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.803210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.803121 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-model-cache\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.809551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.809523 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"premium-w1-e19850-kserve-self-signed-certs\"" Nov 03 22:23:39.811863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.811838 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk"] Nov 03 22:23:39.903655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.903618 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-tmp-dir\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.903655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.903652 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2b69ca-544e-4521-97a2-09a6cecf868a-tls-certs\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.903896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.903715 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-home\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.903896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.903792 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-dshm\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.903896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.903824 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p742\" (UniqueName: \"kubernetes.io/projected/2d2b69ca-544e-4521-97a2-09a6cecf868a-kube-api-access-4p742\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.903896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.903861 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-model-cache\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.904227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.904207 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-home\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.904346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.904327 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-model-cache\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.904411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.904377 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-tmp-dir\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.905993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.905971 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-dshm\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.906247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.906231 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2b69ca-544e-4521-97a2-09a6cecf868a-tls-certs\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:39.925907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:39.925886 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p742\" (UniqueName: \"kubernetes.io/projected/2d2b69ca-544e-4521-97a2-09a6cecf868a-kube-api-access-4p742\") pod \"premium-w1-e19850-kserve-59bcc8c6d7-xzcnk\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:40.097468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:40.097380 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:40.395710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:40.395686 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk"] Nov 03 22:23:40.397469 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:23:40.397439 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d2b69ca_544e_4521_97a2_09a6cecf868a.slice/crio-f39785c9edb40171438d78a9abd576439580ad9583d4ab0f2f7ac7406ac82b1c WatchSource:0}: Error finding container f39785c9edb40171438d78a9abd576439580ad9583d4ab0f2f7ac7406ac82b1c: Status 404 returned error can't find the container with id f39785c9edb40171438d78a9abd576439580ad9583d4ab0f2f7ac7406ac82b1c Nov 03 22:23:40.795086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:40.795050 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" event={"ID":"2d2b69ca-544e-4521-97a2-09a6cecf868a","Type":"ContainerStarted","Data":"47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2"} Nov 03 22:23:40.795565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:40.795095 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" event={"ID":"2d2b69ca-544e-4521-97a2-09a6cecf868a","Type":"ContainerStarted","Data":"f39785c9edb40171438d78a9abd576439580ad9583d4ab0f2f7ac7406ac82b1c"} Nov 03 22:23:40.832268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:40.832216 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podStartSLOduration=1.832203812 podStartE2EDuration="1.832203812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:40.831339276 +0000 UTC m=+647.235280245" watchObservedRunningTime="2026-11-03 22:23:40.832203812 +0000 UTC m=+647.236144787" Nov 03 22:23:41.098255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:41.098180 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:44.050044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:44.049988 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:23:46.498335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.498191 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8"] Nov 03 22:23:46.503807 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.503782 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.526271 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.526239 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"premium-w5-434ce3-kserve-self-signed-certs\"" Nov 03 22:23:46.528831 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.528803 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8"] Nov 03 22:23:46.578130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.578089 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-tmp-dir\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.578130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.578130 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-dshm\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.578385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.578162 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qks9n\" (UniqueName: \"kubernetes.io/projected/5e2730f7-7e63-496a-9155-76818726faff-kube-api-access-qks9n\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.578385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.578327 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-home\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.578385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.578369 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2730f7-7e63-496a-9155-76818726faff-tls-certs\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.578497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.578459 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-model-cache\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.679226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.679187 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-dshm\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.679455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.679240 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qks9n\" (UniqueName: \"kubernetes.io/projected/5e2730f7-7e63-496a-9155-76818726faff-kube-api-access-qks9n\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.679455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.679275 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-home\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.679455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.679333 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2730f7-7e63-496a-9155-76818726faff-tls-certs\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.679455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.679378 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-model-cache\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.679681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.679487 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-tmp-dir\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.679792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.679767 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-home\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.679792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.679784 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-model-cache\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.679968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.679842 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-tmp-dir\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.681588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.681561 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-dshm\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.681982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.681960 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2730f7-7e63-496a-9155-76818726faff-tls-certs\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.708800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.708765 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qks9n\" (UniqueName: \"kubernetes.io/projected/5e2730f7-7e63-496a-9155-76818726faff-kube-api-access-qks9n\") pod \"premium-w5-434ce3-kserve-6f6bc9669-p8fh8\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:46.818450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:46.818359 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:47.131029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:47.130953 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8"] Nov 03 22:23:47.134998 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:23:47.134970 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e2730f7_7e63_496a_9155_76818726faff.slice/crio-41bb3bac3e70f153bb28cc2a29b17bfe47f07a7cb2650b1b6424aca2c6f87a20 WatchSource:0}: Error finding container 41bb3bac3e70f153bb28cc2a29b17bfe47f07a7cb2650b1b6424aca2c6f87a20: Status 404 returned error can't find the container with id 41bb3bac3e70f153bb28cc2a29b17bfe47f07a7cb2650b1b6424aca2c6f87a20 Nov 03 22:23:47.837756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:47.837723 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" event={"ID":"5e2730f7-7e63-496a-9155-76818726faff","Type":"ContainerStarted","Data":"7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9"} Nov 03 22:23:47.837756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:47.837765 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" event={"ID":"5e2730f7-7e63-496a-9155-76818726faff","Type":"ContainerStarted","Data":"41bb3bac3e70f153bb28cc2a29b17bfe47f07a7cb2650b1b6424aca2c6f87a20"} Nov 03 22:23:47.875238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:47.875189 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podStartSLOduration=1.875175208 podStartE2EDuration="1.875175208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:47.874148014 +0000 UTC m=+654.278088989" watchObservedRunningTime="2026-11-03 22:23:47.875175208 +0000 UTC m=+654.279116183" Nov 03 22:23:48.384218 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:23:48.384184 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:23:48.818605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:48.818578 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:50.098281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:50.098239 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:50.110803 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:50.110773 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:50.865681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:50.865650 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:23:55.548158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.548116 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf"] Nov 03 22:23:55.553903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.553875 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.567163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.566989 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"kube-root-ca.crt\"" Nov 03 22:23:55.567163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.567083 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"default-dockercfg-jvvp5\"" Nov 03 22:23:55.577692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.577652 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"premium-w2-553753-kserve-self-signed-certs\"" Nov 03 22:23:55.577857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.577788 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"openshift-service-ca.crt\"" Nov 03 22:23:55.582459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.582432 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf"] Nov 03 22:23:55.701124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.701091 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2778075f-b2e9-4e82-b8ac-7489c9a55918-tls-certs\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.701340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.701131 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngx5\" (UniqueName: \"kubernetes.io/projected/2778075f-b2e9-4e82-b8ac-7489c9a55918-kube-api-access-cngx5\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.701340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.701257 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-model-cache\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.701464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.701448 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-tmp-dir\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.701580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.701560 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-home\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.701640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.701624 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-dshm\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.802766 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.802667 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2778075f-b2e9-4e82-b8ac-7489c9a55918-tls-certs\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.802766 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.802723 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngx5\" (UniqueName: \"kubernetes.io/projected/2778075f-b2e9-4e82-b8ac-7489c9a55918-kube-api-access-cngx5\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.803011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.802805 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-model-cache\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.803011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.802966 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-tmp-dir\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.803134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.803070 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-home\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.803364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.803335 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-dshm\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.803745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.803717 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-tmp-dir\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.803844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.803739 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-model-cache\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.803844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.803768 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-home\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.806072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.806049 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2778075f-b2e9-4e82-b8ac-7489c9a55918-tls-certs\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.806165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.806143 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-dshm\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.824100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.824070 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngx5\" (UniqueName: \"kubernetes.io/projected/2778075f-b2e9-4e82-b8ac-7489c9a55918-kube-api-access-cngx5\") pod \"premium-w2-553753-kserve-59b6d6799b-z57gf\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:55.885335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:55.885307 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:23:56.189438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:56.189376 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf"] Nov 03 22:23:56.192403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:23:56.192377 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2778075f_b2e9_4e82_b8ac_7489c9a55918.slice/crio-3d08d0c35e85e2351350dd7ab889ab3e0e36ea8177bdac995942031f3cbb8cd9 WatchSource:0}: Error finding container 3d08d0c35e85e2351350dd7ab889ab3e0e36ea8177bdac995942031f3cbb8cd9: Status 404 returned error can't find the container with id 3d08d0c35e85e2351350dd7ab889ab3e0e36ea8177bdac995942031f3cbb8cd9 Nov 03 22:23:56.819485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:56.819448 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:56.832561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:56.832534 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:56.893630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:56.893591 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" event={"ID":"2778075f-b2e9-4e82-b8ac-7489c9a55918","Type":"ContainerStarted","Data":"4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e"} Nov 03 22:23:56.893792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:56.893642 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" event={"ID":"2778075f-b2e9-4e82-b8ac-7489c9a55918","Type":"ContainerStarted","Data":"3d08d0c35e85e2351350dd7ab889ab3e0e36ea8177bdac995942031f3cbb8cd9"} Nov 03 22:23:56.904769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:56.904746 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:23:56.934612 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:56.934552 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podStartSLOduration=1.934534566 podStartE2EDuration="1.934534566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:23:56.93278002 +0000 UTC m=+663.336720994" watchObservedRunningTime="2026-11-03 22:23:56.934534566 +0000 UTC m=+663.338475542" Nov 03 22:23:57.885405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:23:57.885371 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:24:04.235482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:04.235438 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:24:05.886353 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:05.886317 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:24:05.899808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:05.899781 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:24:05.964338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:05.964307 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:24:18.385198 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:24:18.385157 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:24:20.008519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.008475 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb"] Nov 03 22:24:20.013599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.013573 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.026193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.026109 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:24:20.029914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.029887 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb"] Nov 03 22:24:20.036931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.036904 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:24:20.037075 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.037057 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:24:20.037953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.037932 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:24:20.120191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.120156 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef49c761-vwnnb\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.120387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.120226 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef49c761-vwnnb\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.120387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.120349 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwk64\" (UniqueName: \"kubernetes.io/projected/ca28ac88-227f-4a7c-84bb-78701735a634-kube-api-access-nwk64\") pod \"maas-api-revoke-sub-ef49c761-vwnnb\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.221971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.221933 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef49c761-vwnnb\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.222146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.222015 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef49c761-vwnnb\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.222146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.222119 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwk64\" (UniqueName: \"kubernetes.io/projected/ca28ac88-227f-4a7c-84bb-78701735a634-kube-api-access-nwk64\") pod \"maas-api-revoke-sub-ef49c761-vwnnb\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.222629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.222606 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef49c761-vwnnb\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.224342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.224322 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef49c761-vwnnb\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.243145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.243118 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwk64\" (UniqueName: \"kubernetes.io/projected/ca28ac88-227f-4a7c-84bb-78701735a634-kube-api-access-nwk64\") pod \"maas-api-revoke-sub-ef49c761-vwnnb\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.328671 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.328586 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:20.630110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.629886 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb"] Nov 03 22:24:20.632288 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:24:20.632259 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca28ac88_227f_4a7c_84bb_78701735a634.slice/crio-b74392346ea9777ee1050905c6e5769dd87e18af8465a3ec262228eda1dff12d WatchSource:0}: Error finding container b74392346ea9777ee1050905c6e5769dd87e18af8465a3ec262228eda1dff12d: Status 404 returned error can't find the container with id b74392346ea9777ee1050905c6e5769dd87e18af8465a3ec262228eda1dff12d Nov 03 22:24:20.634769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:20.634751 4903 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:24:21.048560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:21.048526 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" event={"ID":"ca28ac88-227f-4a7c-84bb-78701735a634","Type":"ContainerStarted","Data":"b74392346ea9777ee1050905c6e5769dd87e18af8465a3ec262228eda1dff12d"} Nov 03 22:24:27.673161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.673125 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d"] Nov 03 22:24:27.678575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.678554 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.706385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.706356 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-553753-openshift-default-dockercfg-vgskd\"" Nov 03 22:24:27.709974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.709951 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d"] Nov 03 22:24:27.807859 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.807823 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.807859 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.807862 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.808163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.808037 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.808163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.808091 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczqd\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-kube-api-access-dczqd\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.808241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.808167 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.808241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.808203 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.808335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.808248 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.808335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.808289 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.808404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.808363 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.909634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.909605 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.909799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.909641 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.909799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.909761 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.909921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.909822 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.909921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.909870 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.909921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.909888 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.909921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.909906 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.910135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.910010 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.910135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.910029 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.910135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.910078 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczqd\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-kube-api-access-dczqd\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.910135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.910124 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.910387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.910226 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.910387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.910277 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.910511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.910448 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.912182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.912157 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.912408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.912392 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.930618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.930567 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.930701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.930648 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczqd\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-kube-api-access-dczqd\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-ktl7d\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:27.993247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:27.993216 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:28.318706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:28.318675 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d"] Nov 03 22:24:28.320653 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:24:28.320627 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda167e1e8_f6f8_4df5_87e6_5fc3d8d3c575.slice/crio-f4a2eba08fda798b70fcdaf8f04912d28d94a47de93fadaae3e329b6a371ac42 WatchSource:0}: Error finding container f4a2eba08fda798b70fcdaf8f04912d28d94a47de93fadaae3e329b6a371ac42: Status 404 returned error can't find the container with id f4a2eba08fda798b70fcdaf8f04912d28d94a47de93fadaae3e329b6a371ac42 Nov 03 22:24:28.322888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:28.322855 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:24:28.322995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:28.322935 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:24:28.322995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:28.322982 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:24:29.104559 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:29.104529 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" event={"ID":"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575","Type":"ContainerStarted","Data":"7f951ce8f58e0a974472f7d8d9b5ae98005d4b9443ce9befeee085a31e7be08a"} Nov 03 22:24:29.104922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:29.104579 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" event={"ID":"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575","Type":"ContainerStarted","Data":"f4a2eba08fda798b70fcdaf8f04912d28d94a47de93fadaae3e329b6a371ac42"} Nov 03 22:24:29.169482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:29.169426 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" podStartSLOduration=2.169410727 podStartE2EDuration="2.169410727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:29.167730795 +0000 UTC m=+695.571671770" watchObservedRunningTime="2026-11-03 22:24:29.169410727 +0000 UTC m=+695.573351784" Nov 03 22:24:29.993587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:29.993556 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:29.995781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:29.995751 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:29.995921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:29.995803 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:30.994249 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:30.994205 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:30.994740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:30.994273 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:31.994432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:31.994403 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:31.994771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:31.994447 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:32.124404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:32.124364 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" event={"ID":"ca28ac88-227f-4a7c-84bb-78701735a634","Type":"ContainerStarted","Data":"3df1d1c301d011e8ef538040c32c01c9bd60ad5081558c91c219981c6b09de28"} Nov 03 22:24:32.207561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:32.207505 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" podStartSLOduration=2.313806789 podStartE2EDuration="13.207481548s" totalImagesPullingTime="10.893674759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:24:32.205823134 +0000 UTC m=+698.609764109" watchObservedRunningTime="2026-11-03 22:24:32.207481548 +0000 UTC m=+698.611422522" Nov 03 22:24:32.994521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:32.994425 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:32.994521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:32.994485 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:33.132715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:33.132682 4903 generic.go:356] "Generic (PLEG): container finished" podID="ca28ac88-227f-4a7c-84bb-78701735a634" containerID="3df1d1c301d011e8ef538040c32c01c9bd60ad5081558c91c219981c6b09de28" exitCode=0 Nov 03 22:24:33.132715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:33.132718 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" event={"ID":"ca28ac88-227f-4a7c-84bb-78701735a634","Type":"ContainerDied","Data":"3df1d1c301d011e8ef538040c32c01c9bd60ad5081558c91c219981c6b09de28"} Nov 03 22:24:33.993881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:33.993849 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 22:24:33.994067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:33.993913 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 22:24:34.313737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.313705 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:34.420811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.420780 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-xks-ca\") pod \"ca28ac88-227f-4a7c-84bb-78701735a634\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " Nov 03 22:24:34.420971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.420881 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca28ac88-227f-4a7c-84bb-78701735a634-kube-api-access-nwk64\" (UniqueName: \"kubernetes.io/projected/ca28ac88-227f-4a7c-84bb-78701735a634-kube-api-access-nwk64\") pod \"ca28ac88-227f-4a7c-84bb-78701735a634\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " Nov 03 22:24:34.420971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.420919 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-openshift-ca\") pod \"ca28ac88-227f-4a7c-84bb-78701735a634\" (UID: \"ca28ac88-227f-4a7c-84bb-78701735a634\") " Nov 03 22:24:34.421341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.421284 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-openshift-ca" pod "ca28ac88-227f-4a7c-84bb-78701735a634" (UID: "ca28ac88-227f-4a7c-84bb-78701735a634"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:24:34.423142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.423114 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-xks-ca" pod "ca28ac88-227f-4a7c-84bb-78701735a634" (UID: "ca28ac88-227f-4a7c-84bb-78701735a634"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:24:34.423232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.423206 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca28ac88-227f-4a7c-84bb-78701735a634-kube-api-access-nwk64" pod "ca28ac88-227f-4a7c-84bb-78701735a634" (UID: "ca28ac88-227f-4a7c-84bb-78701735a634"). InnerVolumeSpecName "kube-api-access-nwk64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:24:34.523781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.523738 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:24:34.523781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.523776 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwk64\" (UniqueName: \"kubernetes.io/projected/ca28ac88-227f-4a7c-84bb-78701735a634-kube-api-access-nwk64\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:24:34.523954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.523790 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca28ac88-227f-4a7c-84bb-78701735a634-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:24:34.998032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.998001 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:34.998307 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.998275 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:34.998875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:34.998855 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:24:35.147207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:35.147174 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" event={"ID":"ca28ac88-227f-4a7c-84bb-78701735a634","Type":"ContainerDied","Data":"b74392346ea9777ee1050905c6e5769dd87e18af8465a3ec262228eda1dff12d"} Nov 03 22:24:35.147408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:35.147212 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb" Nov 03 22:24:35.147408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:35.147227 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b74392346ea9777ee1050905c6e5769dd87e18af8465a3ec262228eda1dff12d" Nov 03 22:24:48.384139 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:24:48.384103 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:24:50.064436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:24:50.064403 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:25:08.611810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.611775 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5"] Nov 03 22:25:08.612662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.612636 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ca28ac88-227f-4a7c-84bb-78701735a634" containerName="revoke-keys" Nov 03 22:25:08.612662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.612659 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca28ac88-227f-4a7c-84bb-78701735a634" containerName="revoke-keys" Nov 03 22:25:08.612889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.612786 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca28ac88-227f-4a7c-84bb-78701735a634" containerName="revoke-keys" Nov 03 22:25:08.620756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.620733 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.642620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.642548 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5"] Nov 03 22:25:08.646630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.646606 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:25:08.646867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.646837 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:25:08.647026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.647003 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:25:08.648512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.648493 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:25:08.716523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.716490 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qwf6\" (UniqueName: \"kubernetes.io/projected/c043b1b2-b329-4a63-b92f-62a82c1841cf-kube-api-access-9qwf6\") pod \"maas-api-revoke-sub-987e6ab4-5ptx5\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.716721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.716554 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987e6ab4-5ptx5\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.716796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.716678 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987e6ab4-5ptx5\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.825045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.825003 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qwf6\" (UniqueName: \"kubernetes.io/projected/c043b1b2-b329-4a63-b92f-62a82c1841cf-kube-api-access-9qwf6\") pod \"maas-api-revoke-sub-987e6ab4-5ptx5\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.826154 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.826128 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987e6ab4-5ptx5\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.826396 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.826382 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987e6ab4-5ptx5\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.827429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.827405 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987e6ab4-5ptx5\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.831114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.831086 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987e6ab4-5ptx5\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.849998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.849964 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qwf6\" (UniqueName: \"kubernetes.io/projected/c043b1b2-b329-4a63-b92f-62a82c1841cf-kube-api-access-9qwf6\") pod \"maas-api-revoke-sub-987e6ab4-5ptx5\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:08.949319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:08.946498 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:09.334498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:09.334403 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5"] Nov 03 22:25:10.384190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:10.384147 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" event={"ID":"c043b1b2-b329-4a63-b92f-62a82c1841cf","Type":"ContainerStarted","Data":"597c9f0c698880cca3cf8bb6ac22a6c416e354d8cf06d91bd7fe72816ce1d38b"} Nov 03 22:25:10.384713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:10.384207 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" event={"ID":"c043b1b2-b329-4a63-b92f-62a82c1841cf","Type":"ContainerStarted","Data":"234a1125c80cc25b041b47dc465f1a2d27837e3b2044e93599b05dd7ee7f0b32"} Nov 03 22:25:10.425278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:10.425218 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" podStartSLOduration=2.425203778 podStartE2EDuration="2.425203778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:10.421991654 +0000 UTC m=+736.825932635" watchObservedRunningTime="2026-11-03 22:25:10.425203778 +0000 UTC m=+736.829144753" Nov 03 22:25:11.393491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:11.393453 4903 generic.go:356] "Generic (PLEG): container finished" podID="c043b1b2-b329-4a63-b92f-62a82c1841cf" containerID="597c9f0c698880cca3cf8bb6ac22a6c416e354d8cf06d91bd7fe72816ce1d38b" exitCode=0 Nov 03 22:25:11.393993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:11.393534 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" event={"ID":"c043b1b2-b329-4a63-b92f-62a82c1841cf","Type":"ContainerDied","Data":"597c9f0c698880cca3cf8bb6ac22a6c416e354d8cf06d91bd7fe72816ce1d38b"} Nov 03 22:25:12.597379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.597352 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:12.743055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.742963 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-openshift-ca\") pod \"c043b1b2-b329-4a63-b92f-62a82c1841cf\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " Nov 03 22:25:12.743055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.743048 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-xks-ca\") pod \"c043b1b2-b329-4a63-b92f-62a82c1841cf\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " Nov 03 22:25:12.743332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.743124 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c043b1b2-b329-4a63-b92f-62a82c1841cf-kube-api-access-9qwf6\" (UniqueName: \"kubernetes.io/projected/c043b1b2-b329-4a63-b92f-62a82c1841cf-kube-api-access-9qwf6\") pod \"c043b1b2-b329-4a63-b92f-62a82c1841cf\" (UID: \"c043b1b2-b329-4a63-b92f-62a82c1841cf\") " Nov 03 22:25:12.744507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.744473 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-openshift-ca" pod "c043b1b2-b329-4a63-b92f-62a82c1841cf" (UID: "c043b1b2-b329-4a63-b92f-62a82c1841cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:12.745758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.745730 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c043b1b2-b329-4a63-b92f-62a82c1841cf-kube-api-access-9qwf6" pod "c043b1b2-b329-4a63-b92f-62a82c1841cf" (UID: "c043b1b2-b329-4a63-b92f-62a82c1841cf"). InnerVolumeSpecName "kube-api-access-9qwf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:12.746045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.746022 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-xks-ca" pod "c043b1b2-b329-4a63-b92f-62a82c1841cf" (UID: "c043b1b2-b329-4a63-b92f-62a82c1841cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:12.844707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.844668 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:12.844707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.844703 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c043b1b2-b329-4a63-b92f-62a82c1841cf-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:12.844707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:12.844716 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qwf6\" (UniqueName: \"kubernetes.io/projected/c043b1b2-b329-4a63-b92f-62a82c1841cf-kube-api-access-9qwf6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:13.408950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:13.408923 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" Nov 03 22:25:13.409123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:13.408921 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5" event={"ID":"c043b1b2-b329-4a63-b92f-62a82c1841cf","Type":"ContainerDied","Data":"234a1125c80cc25b041b47dc465f1a2d27837e3b2044e93599b05dd7ee7f0b32"} Nov 03 22:25:13.409123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:13.409052 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="234a1125c80cc25b041b47dc465f1a2d27837e3b2044e93599b05dd7ee7f0b32" Nov 03 22:25:14.055685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.055650 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl"] Nov 03 22:25:14.056275 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.056247 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c043b1b2-b329-4a63-b92f-62a82c1841cf" containerName="revoke-keys" Nov 03 22:25:14.056275 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.056263 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c043b1b2-b329-4a63-b92f-62a82c1841cf" containerName="revoke-keys" Nov 03 22:25:14.056475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.056455 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c043b1b2-b329-4a63-b92f-62a82c1841cf" containerName="revoke-keys" Nov 03 22:25:14.060634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.060611 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.073714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.073681 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:25:14.078759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.076908 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl"] Nov 03 22:25:14.083683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.083659 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:25:14.083857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.083718 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:25:14.083857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.083792 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:25:14.163617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.163580 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b0790a-dpgjl\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.163802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.163680 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj94x\" (UniqueName: \"kubernetes.io/projected/ff4e0be2-8036-422b-8be4-0b904c0e0334-kube-api-access-cj94x\") pod \"maas-api-revoke-sub-b6b0790a-dpgjl\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.163845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.163798 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b0790a-dpgjl\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.265487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.265443 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj94x\" (UniqueName: \"kubernetes.io/projected/ff4e0be2-8036-422b-8be4-0b904c0e0334-kube-api-access-cj94x\") pod \"maas-api-revoke-sub-b6b0790a-dpgjl\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.265687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.265530 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b0790a-dpgjl\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.265687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.265658 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b0790a-dpgjl\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.266195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.266164 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b0790a-dpgjl\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.268166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.268133 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b0790a-dpgjl\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.286486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.286460 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj94x\" (UniqueName: \"kubernetes.io/projected/ff4e0be2-8036-422b-8be4-0b904c0e0334-kube-api-access-cj94x\") pod \"maas-api-revoke-sub-b6b0790a-dpgjl\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.345180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.345086 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz"] Nov 03 22:25:14.350062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.350035 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.368531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.368501 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz"] Nov 03 22:25:14.382096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.382060 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:14.469948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.469901 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2020770c-wvsjz\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.470094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.470023 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2020770c-wvsjz\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.470140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.470110 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjw7\" (UniqueName: \"kubernetes.io/projected/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-kube-api-access-snjw7\") pod \"maas-api-revoke-sub-2020770c-wvsjz\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.570836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.570773 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2020770c-wvsjz\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.571003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.570875 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2020770c-wvsjz\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.571003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.570920 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjw7\" (UniqueName: \"kubernetes.io/projected/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-kube-api-access-snjw7\") pod \"maas-api-revoke-sub-2020770c-wvsjz\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.571606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.571582 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2020770c-wvsjz\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.573368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.573343 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2020770c-wvsjz\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.592020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.591993 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjw7\" (UniqueName: \"kubernetes.io/projected/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-kube-api-access-snjw7\") pod \"maas-api-revoke-sub-2020770c-wvsjz\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.670591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.670559 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:14.698771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:14.697919 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl"] Nov 03 22:25:15.015720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:15.015691 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz"] Nov 03 22:25:15.047280 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:15.047240 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2d660a_a0b6_43f7_bd95_425c8b7fe7b0.slice/crio-c4e751682d0f1a31fdd6140a25d3659dfc3051aff4c7b7e60c1436723ee5b3da WatchSource:0}: Error finding container c4e751682d0f1a31fdd6140a25d3659dfc3051aff4c7b7e60c1436723ee5b3da: Status 404 returned error can't find the container with id c4e751682d0f1a31fdd6140a25d3659dfc3051aff4c7b7e60c1436723ee5b3da Nov 03 22:25:15.447361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:15.447326 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" event={"ID":"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0","Type":"ContainerStarted","Data":"c4e751682d0f1a31fdd6140a25d3659dfc3051aff4c7b7e60c1436723ee5b3da"} Nov 03 22:25:15.450464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:15.450429 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" event={"ID":"ff4e0be2-8036-422b-8be4-0b904c0e0334","Type":"ContainerStarted","Data":"1d915e4ff8b5e29824984cf7ab7fa4a7b498265472d8a1891eb0951a814e956b"} Nov 03 22:25:15.450603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:15.450496 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" event={"ID":"ff4e0be2-8036-422b-8be4-0b904c0e0334","Type":"ContainerStarted","Data":"f928fd42b501025496e1c169820af66d47ae1cfae1152daca6fc05085d313cff"} Nov 03 22:25:15.489314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:15.488423 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" podStartSLOduration=1.4884048189999999 podStartE2EDuration="1.488404819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:15.48758898 +0000 UTC m=+741.891529956" watchObservedRunningTime="2026-11-03 22:25:15.488404819 +0000 UTC m=+741.892345796" Nov 03 22:25:15.912323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:15.912259 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n"] Nov 03 22:25:15.916337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:15.916280 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:15.935656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:15.935623 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n"] Nov 03 22:25:16.095204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.095176 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x4pb\" (UniqueName: \"kubernetes.io/projected/3fe7a4d4-ec2f-4024-ac21-03e677210362-kube-api-access-5x4pb\") pod \"maas-api-revoke-sub-75577660-78s6n\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.095366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.095234 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75577660-78s6n\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.095517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.095492 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75577660-78s6n\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.196364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.196275 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75577660-78s6n\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.196513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.196378 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x4pb\" (UniqueName: \"kubernetes.io/projected/3fe7a4d4-ec2f-4024-ac21-03e677210362-kube-api-access-5x4pb\") pod \"maas-api-revoke-sub-75577660-78s6n\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.196513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.196450 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75577660-78s6n\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.196977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.196953 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75577660-78s6n\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.198901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.198879 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75577660-78s6n\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.218115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.218086 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x4pb\" (UniqueName: \"kubernetes.io/projected/3fe7a4d4-ec2f-4024-ac21-03e677210362-kube-api-access-5x4pb\") pod \"maas-api-revoke-sub-75577660-78s6n\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.232447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.232418 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:16.459323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.459226 4903 generic.go:356] "Generic (PLEG): container finished" podID="bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0" containerID="cb343e68b515a1681fd9d136fdee97c6393b6b64347b06f4c8b576ea78f5fe6a" exitCode=0 Nov 03 22:25:16.459731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.459326 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" event={"ID":"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0","Type":"ContainerDied","Data":"cb343e68b515a1681fd9d136fdee97c6393b6b64347b06f4c8b576ea78f5fe6a"} Nov 03 22:25:16.461336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.461307 4903 generic.go:356] "Generic (PLEG): container finished" podID="ff4e0be2-8036-422b-8be4-0b904c0e0334" containerID="1d915e4ff8b5e29824984cf7ab7fa4a7b498265472d8a1891eb0951a814e956b" exitCode=0 Nov 03 22:25:16.461454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.461390 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" event={"ID":"ff4e0be2-8036-422b-8be4-0b904c0e0334","Type":"ContainerDied","Data":"1d915e4ff8b5e29824984cf7ab7fa4a7b498265472d8a1891eb0951a814e956b"} Nov 03 22:25:16.540587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:16.540543 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n"] Nov 03 22:25:16.542316 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:16.542269 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe7a4d4_ec2f_4024_ac21_03e677210362.slice/crio-5a13bf786b4c08fc766b2deabd32cf13077f9aa97e0daa6018455bd844b108a0 WatchSource:0}: Error finding container 5a13bf786b4c08fc766b2deabd32cf13077f9aa97e0daa6018455bd844b108a0: Status 404 returned error can't find the container with id 5a13bf786b4c08fc766b2deabd32cf13077f9aa97e0daa6018455bd844b108a0 Nov 03 22:25:17.474313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.471379 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" event={"ID":"3fe7a4d4-ec2f-4024-ac21-03e677210362","Type":"ContainerStarted","Data":"78f18bcf4fd51d6fd680b322c36ce02d5aefd349e62d717b6c8e61080423e2c0"} Nov 03 22:25:17.474313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.471437 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" event={"ID":"3fe7a4d4-ec2f-4024-ac21-03e677210362","Type":"ContainerStarted","Data":"5a13bf786b4c08fc766b2deabd32cf13077f9aa97e0daa6018455bd844b108a0"} Nov 03 22:25:17.675320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.660355 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:17.716607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.716525 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:17.727778 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.727738 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-xks-ca\") pod \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " Nov 03 22:25:17.727924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.727824 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-kube-api-access-snjw7\" (UniqueName: \"kubernetes.io/projected/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-kube-api-access-snjw7\") pod \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " Nov 03 22:25:17.727924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.727874 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4e0be2-8036-422b-8be4-0b904c0e0334-kube-api-access-cj94x\" (UniqueName: \"kubernetes.io/projected/ff4e0be2-8036-422b-8be4-0b904c0e0334-kube-api-access-cj94x\") pod \"ff4e0be2-8036-422b-8be4-0b904c0e0334\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " Nov 03 22:25:17.728045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.728034 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-openshift-ca\") pod \"ff4e0be2-8036-422b-8be4-0b904c0e0334\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " Nov 03 22:25:17.728098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.728081 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-openshift-ca\") pod \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\" (UID: \"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0\") " Nov 03 22:25:17.728152 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.728128 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-xks-ca\") pod \"ff4e0be2-8036-422b-8be4-0b904c0e0334\" (UID: \"ff4e0be2-8036-422b-8be4-0b904c0e0334\") " Nov 03 22:25:17.730073 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.729983 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-openshift-ca" pod "bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0" (UID: "bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:17.730384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.730350 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-openshift-ca" pod "ff4e0be2-8036-422b-8be4-0b904c0e0334" (UID: "ff4e0be2-8036-422b-8be4-0b904c0e0334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:17.731900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.731862 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-kube-api-access-snjw7" pod "bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0" (UID: "bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0"). InnerVolumeSpecName "kube-api-access-snjw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:17.736840 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.736503 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-xks-ca" pod "ff4e0be2-8036-422b-8be4-0b904c0e0334" (UID: "ff4e0be2-8036-422b-8be4-0b904c0e0334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:17.743200 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.738424 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-xks-ca" pod "bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0" (UID: "bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:17.747313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.746363 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4e0be2-8036-422b-8be4-0b904c0e0334-kube-api-access-cj94x" pod "ff4e0be2-8036-422b-8be4-0b904c0e0334" (UID: "ff4e0be2-8036-422b-8be4-0b904c0e0334"). InnerVolumeSpecName "kube-api-access-cj94x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:17.829197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.829153 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:17.829197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.829192 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:17.829434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.829207 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4e0be2-8036-422b-8be4-0b904c0e0334-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:17.829434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.829220 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:17.829434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.829231 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snjw7\" (UniqueName: \"kubernetes.io/projected/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0-kube-api-access-snjw7\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:17.829434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:17.829244 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj94x\" (UniqueName: \"kubernetes.io/projected/ff4e0be2-8036-422b-8be4-0b904c0e0334-kube-api-access-cj94x\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:18.453321 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:25:18.450306 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:25:18.510029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.509625 4903 generic.go:356] "Generic (PLEG): container finished" podID="3fe7a4d4-ec2f-4024-ac21-03e677210362" containerID="78f18bcf4fd51d6fd680b322c36ce02d5aefd349e62d717b6c8e61080423e2c0" exitCode=0 Nov 03 22:25:18.510029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.509721 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" event={"ID":"3fe7a4d4-ec2f-4024-ac21-03e677210362","Type":"ContainerDied","Data":"78f18bcf4fd51d6fd680b322c36ce02d5aefd349e62d717b6c8e61080423e2c0"} Nov 03 22:25:18.515257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.515227 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" event={"ID":"bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0","Type":"ContainerDied","Data":"c4e751682d0f1a31fdd6140a25d3659dfc3051aff4c7b7e60c1436723ee5b3da"} Nov 03 22:25:18.515455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.515428 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4e751682d0f1a31fdd6140a25d3659dfc3051aff4c7b7e60c1436723ee5b3da" Nov 03 22:25:18.517579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.517560 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" event={"ID":"ff4e0be2-8036-422b-8be4-0b904c0e0334","Type":"ContainerDied","Data":"f928fd42b501025496e1c169820af66d47ae1cfae1152daca6fc05085d313cff"} Nov 03 22:25:18.517751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.517735 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f928fd42b501025496e1c169820af66d47ae1cfae1152daca6fc05085d313cff" Nov 03 22:25:18.517876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.515613 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz" Nov 03 22:25:18.521011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.517435 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl" Nov 03 22:25:18.833485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.833166 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:18.875931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.875348 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe7a4d4-ec2f-4024-ac21-03e677210362-kube-api-access-5x4pb\" (UniqueName: \"kubernetes.io/projected/3fe7a4d4-ec2f-4024-ac21-03e677210362-kube-api-access-5x4pb\") pod \"3fe7a4d4-ec2f-4024-ac21-03e677210362\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " Nov 03 22:25:18.875931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.875422 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-xks-ca\") pod \"3fe7a4d4-ec2f-4024-ac21-03e677210362\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " Nov 03 22:25:18.875931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.875520 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-openshift-ca\") pod \"3fe7a4d4-ec2f-4024-ac21-03e677210362\" (UID: \"3fe7a4d4-ec2f-4024-ac21-03e677210362\") " Nov 03 22:25:18.876208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.876017 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-openshift-ca" pod "3fe7a4d4-ec2f-4024-ac21-03e677210362" (UID: "3fe7a4d4-ec2f-4024-ac21-03e677210362"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:18.876471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.876451 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:18.905326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.899673 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe7a4d4-ec2f-4024-ac21-03e677210362-kube-api-access-5x4pb" pod "3fe7a4d4-ec2f-4024-ac21-03e677210362" (UID: "3fe7a4d4-ec2f-4024-ac21-03e677210362"). InnerVolumeSpecName "kube-api-access-5x4pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:18.915792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.914426 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-xks-ca" pod "3fe7a4d4-ec2f-4024-ac21-03e677210362" (UID: "3fe7a4d4-ec2f-4024-ac21-03e677210362"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:18.977972 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.977936 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe7a4d4-ec2f-4024-ac21-03e677210362-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:18.977972 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:18.977971 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x4pb\" (UniqueName: \"kubernetes.io/projected/3fe7a4d4-ec2f-4024-ac21-03e677210362-kube-api-access-5x4pb\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:19.405134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405100 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb"] Nov 03 22:25:19.405723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405701 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3fe7a4d4-ec2f-4024-ac21-03e677210362" containerName="revoke-keys" Nov 03 22:25:19.405723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405720 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe7a4d4-ec2f-4024-ac21-03e677210362" containerName="revoke-keys" Nov 03 22:25:19.405913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405745 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0" containerName="revoke-keys" Nov 03 22:25:19.405913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405753 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0" containerName="revoke-keys" Nov 03 22:25:19.405913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405776 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ff4e0be2-8036-422b-8be4-0b904c0e0334" containerName="revoke-keys" Nov 03 22:25:19.405913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405784 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4e0be2-8036-422b-8be4-0b904c0e0334" containerName="revoke-keys" Nov 03 22:25:19.405913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405896 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe7a4d4-ec2f-4024-ac21-03e677210362" containerName="revoke-keys" Nov 03 22:25:19.405913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405906 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0" containerName="revoke-keys" Nov 03 22:25:19.405913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.405915 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4e0be2-8036-422b-8be4-0b904c0e0334" containerName="revoke-keys" Nov 03 22:25:19.409562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.409540 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.426766 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.426742 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb"] Nov 03 22:25:19.483917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.483884 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb0e211a-s8ggb\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.484082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.483964 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb0e211a-s8ggb\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.484082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.484060 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctz8s\" (UniqueName: \"kubernetes.io/projected/945da303-35e3-4f29-b55c-b46150e7aa95-kube-api-access-ctz8s\") pod \"maas-api-revoke-sub-fb0e211a-s8ggb\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.525317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.525260 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" Nov 03 22:25:19.525693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.525284 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n" event={"ID":"3fe7a4d4-ec2f-4024-ac21-03e677210362","Type":"ContainerDied","Data":"5a13bf786b4c08fc766b2deabd32cf13077f9aa97e0daa6018455bd844b108a0"} Nov 03 22:25:19.525693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.525372 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a13bf786b4c08fc766b2deabd32cf13077f9aa97e0daa6018455bd844b108a0" Nov 03 22:25:19.585629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.585587 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb0e211a-s8ggb\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.585806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.585693 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb0e211a-s8ggb\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.585806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.585732 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctz8s\" (UniqueName: \"kubernetes.io/projected/945da303-35e3-4f29-b55c-b46150e7aa95-kube-api-access-ctz8s\") pod \"maas-api-revoke-sub-fb0e211a-s8ggb\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.586418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.586391 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb0e211a-s8ggb\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.588003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.587974 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb0e211a-s8ggb\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.606710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.606683 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctz8s\" (UniqueName: \"kubernetes.io/projected/945da303-35e3-4f29-b55c-b46150e7aa95-kube-api-access-ctz8s\") pod \"maas-api-revoke-sub-fb0e211a-s8ggb\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:19.725840 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:19.725746 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:20.034019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:20.033983 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb"] Nov 03 22:25:20.036008 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:20.035982 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945da303_35e3_4f29_b55c_b46150e7aa95.slice/crio-7d61ea3b677891a3e1994ada55d4824bd1e5fb9707c3810b9c3529e14df6ed5d WatchSource:0}: Error finding container 7d61ea3b677891a3e1994ada55d4824bd1e5fb9707c3810b9c3529e14df6ed5d: Status 404 returned error can't find the container with id 7d61ea3b677891a3e1994ada55d4824bd1e5fb9707c3810b9c3529e14df6ed5d Nov 03 22:25:20.532807 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:20.532773 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" event={"ID":"945da303-35e3-4f29-b55c-b46150e7aa95","Type":"ContainerStarted","Data":"079caadab189907d9155f482a04dd7928cf0b76ee2fb1256a541f7a5fa7f11e7"} Nov 03 22:25:20.533265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:20.532829 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" event={"ID":"945da303-35e3-4f29-b55c-b46150e7aa95","Type":"ContainerStarted","Data":"7d61ea3b677891a3e1994ada55d4824bd1e5fb9707c3810b9c3529e14df6ed5d"} Nov 03 22:25:20.571935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:20.571880 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" podStartSLOduration=1.571867711 podStartE2EDuration="1.571867711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:20.569918097 +0000 UTC m=+746.973859073" watchObservedRunningTime="2026-11-03 22:25:20.571867711 +0000 UTC m=+746.975808685" Nov 03 22:25:21.544190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:21.544148 4903 generic.go:356] "Generic (PLEG): container finished" podID="945da303-35e3-4f29-b55c-b46150e7aa95" containerID="079caadab189907d9155f482a04dd7928cf0b76ee2fb1256a541f7a5fa7f11e7" exitCode=0 Nov 03 22:25:21.544676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:21.544249 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" event={"ID":"945da303-35e3-4f29-b55c-b46150e7aa95","Type":"ContainerDied","Data":"079caadab189907d9155f482a04dd7928cf0b76ee2fb1256a541f7a5fa7f11e7"} Nov 03 22:25:23.035521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.035489 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:23.077679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.077641 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-xks-ca\") pod \"945da303-35e3-4f29-b55c-b46150e7aa95\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " Nov 03 22:25:23.077949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.077929 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-openshift-ca\") pod \"945da303-35e3-4f29-b55c-b46150e7aa95\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " Nov 03 22:25:23.078176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.078158 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945da303-35e3-4f29-b55c-b46150e7aa95-kube-api-access-ctz8s\" (UniqueName: \"kubernetes.io/projected/945da303-35e3-4f29-b55c-b46150e7aa95-kube-api-access-ctz8s\") pod \"945da303-35e3-4f29-b55c-b46150e7aa95\" (UID: \"945da303-35e3-4f29-b55c-b46150e7aa95\") " Nov 03 22:25:23.127002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.124794 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-openshift-ca" pod "945da303-35e3-4f29-b55c-b46150e7aa95" (UID: "945da303-35e3-4f29-b55c-b46150e7aa95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:23.131312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.130423 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945da303-35e3-4f29-b55c-b46150e7aa95-kube-api-access-ctz8s" pod "945da303-35e3-4f29-b55c-b46150e7aa95" (UID: "945da303-35e3-4f29-b55c-b46150e7aa95"). InnerVolumeSpecName "kube-api-access-ctz8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:23.135947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.134590 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-xks-ca" pod "945da303-35e3-4f29-b55c-b46150e7aa95" (UID: "945da303-35e3-4f29-b55c-b46150e7aa95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:23.212378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.203139 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:23.212378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.203183 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945da303-35e3-4f29-b55c-b46150e7aa95-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:23.212378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.203202 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctz8s\" (UniqueName: \"kubernetes.io/projected/945da303-35e3-4f29-b55c-b46150e7aa95-kube-api-access-ctz8s\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:23.264318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.264253 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-xt8wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:25:23.264520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.264345 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-xt8wx" podUID="534555a5-ce83-4e60-a28a-b410824419ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:25:23.288316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.284369 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2cwvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:25:23.288316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.284427 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:25:23.613000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.611770 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" event={"ID":"945da303-35e3-4f29-b55c-b46150e7aa95","Type":"ContainerDied","Data":"7d61ea3b677891a3e1994ada55d4824bd1e5fb9707c3810b9c3529e14df6ed5d"} Nov 03 22:25:23.613000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.611873 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d61ea3b677891a3e1994ada55d4824bd1e5fb9707c3810b9c3529e14df6ed5d" Nov 03 22:25:23.613000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.611950 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb" Nov 03 22:25:23.639693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.639657 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl"] Nov 03 22:25:23.640268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.640249 4903 cpu_manager.go:414] "containerMap: removing container" podUID="945da303-35e3-4f29-b55c-b46150e7aa95" containerName="revoke-keys" Nov 03 22:25:23.640395 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.640268 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945da303-35e3-4f29-b55c-b46150e7aa95" containerName="revoke-keys" Nov 03 22:25:23.640510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.640495 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="945da303-35e3-4f29-b55c-b46150e7aa95" containerName="revoke-keys" Nov 03 22:25:23.687341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.686574 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.706318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.703513 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl"] Nov 03 22:25:23.737331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.724098 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec1fb9e2-nlcsl\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.737331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.724263 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec1fb9e2-nlcsl\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.737331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.724375 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55kgc\" (UniqueName: \"kubernetes.io/projected/bc17242f-1ea8-421a-8fc8-c6f784447536-kube-api-access-55kgc\") pod \"maas-api-revoke-sub-ec1fb9e2-nlcsl\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.738587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.738475 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:25:23.738587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.738534 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:25:23.738587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.738559 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:25:23.738587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.738566 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:25:23.835317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.826852 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec1fb9e2-nlcsl\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.835317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.827027 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec1fb9e2-nlcsl\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.835317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.827083 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55kgc\" (UniqueName: \"kubernetes.io/projected/bc17242f-1ea8-421a-8fc8-c6f784447536-kube-api-access-55kgc\") pod \"maas-api-revoke-sub-ec1fb9e2-nlcsl\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.835317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.834380 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec1fb9e2-nlcsl\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.860317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.836716 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2"] Nov 03 22:25:23.860317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.849420 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec1fb9e2-nlcsl\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.860317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.850944 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2"] Nov 03 22:25:23.860317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.851056 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:23.885379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.882746 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55kgc\" (UniqueName: \"kubernetes.io/projected/bc17242f-1ea8-421a-8fc8-c6f784447536-kube-api-access-55kgc\") pod \"maas-api-revoke-sub-ec1fb9e2-nlcsl\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:23.928767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.927906 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78ba40e6-h9qb2\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:23.928767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.928151 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4tw\" (UniqueName: \"kubernetes.io/projected/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-kube-api-access-6d4tw\") pod \"maas-api-revoke-sub-78ba40e6-h9qb2\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:23.928767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:23.928205 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78ba40e6-h9qb2\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:24.028955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.028918 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4tw\" (UniqueName: \"kubernetes.io/projected/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-kube-api-access-6d4tw\") pod \"maas-api-revoke-sub-78ba40e6-h9qb2\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:24.029108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.028992 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78ba40e6-h9qb2\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:24.029108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.029058 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78ba40e6-h9qb2\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:24.029896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.029870 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78ba40e6-h9qb2\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:24.031632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.031611 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78ba40e6-h9qb2\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:24.050195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.050166 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4tw\" (UniqueName: \"kubernetes.io/projected/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-kube-api-access-6d4tw\") pod \"maas-api-revoke-sub-78ba40e6-h9qb2\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:24.111961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.111929 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:24.215993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.215957 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:24.415734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.415708 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl"] Nov 03 22:25:24.416192 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:24.416164 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc17242f_1ea8_421a_8fc8_c6f784447536.slice/crio-e84c8946307fde40c0339300620beb1ce95cfa7f15fa3898e599e9c059f363c0 WatchSource:0}: Error finding container e84c8946307fde40c0339300620beb1ce95cfa7f15fa3898e599e9c059f363c0: Status 404 returned error can't find the container with id e84c8946307fde40c0339300620beb1ce95cfa7f15fa3898e599e9c059f363c0 Nov 03 22:25:24.521065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.521038 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2"] Nov 03 22:25:24.522713 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:24.522691 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be1f7db_bf6a_4455_9c37_b10b499e5b8d.slice/crio-e8740ae3db249f983dba9e150b2621b392e8f57b1d15aa3a7759843a5ad18755 WatchSource:0}: Error finding container e8740ae3db249f983dba9e150b2621b392e8f57b1d15aa3a7759843a5ad18755: Status 404 returned error can't find the container with id e8740ae3db249f983dba9e150b2621b392e8f57b1d15aa3a7759843a5ad18755 Nov 03 22:25:24.629206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.629167 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" event={"ID":"6be1f7db-bf6a-4455-9c37-b10b499e5b8d","Type":"ContainerStarted","Data":"e8740ae3db249f983dba9e150b2621b392e8f57b1d15aa3a7759843a5ad18755"} Nov 03 22:25:24.630450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:24.630425 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" event={"ID":"bc17242f-1ea8-421a-8fc8-c6f784447536","Type":"ContainerStarted","Data":"e84c8946307fde40c0339300620beb1ce95cfa7f15fa3898e599e9c059f363c0"} Nov 03 22:25:25.637539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:25.637446 4903 generic.go:356] "Generic (PLEG): container finished" podID="bc17242f-1ea8-421a-8fc8-c6f784447536" containerID="c849268fbadbb9fe0d461370746f2e4f17073b73b9d1a13a787d367d72250a65" exitCode=0 Nov 03 22:25:25.637974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:25.637531 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" event={"ID":"bc17242f-1ea8-421a-8fc8-c6f784447536","Type":"ContainerDied","Data":"c849268fbadbb9fe0d461370746f2e4f17073b73b9d1a13a787d367d72250a65"} Nov 03 22:25:25.639147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:25.639120 4903 generic.go:356] "Generic (PLEG): container finished" podID="6be1f7db-bf6a-4455-9c37-b10b499e5b8d" containerID="4e7c71364bf28068d687435c09b450e281bc497110f60eae48f4cabaf1c24389" exitCode=0 Nov 03 22:25:25.639270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:25.639201 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" event={"ID":"6be1f7db-bf6a-4455-9c37-b10b499e5b8d","Type":"ContainerDied","Data":"4e7c71364bf28068d687435c09b450e281bc497110f60eae48f4cabaf1c24389"} Nov 03 22:25:26.833920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.833897 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:26.837610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.837589 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:26.861834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.861806 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-xks-ca\") pod \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " Nov 03 22:25:26.861985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.861870 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-openshift-ca\") pod \"bc17242f-1ea8-421a-8fc8-c6f784447536\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " Nov 03 22:25:26.861985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.861905 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-kube-api-access-6d4tw\" (UniqueName: \"kubernetes.io/projected/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-kube-api-access-6d4tw\") pod \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " Nov 03 22:25:26.861985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.861950 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-openshift-ca\") pod \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\" (UID: \"6be1f7db-bf6a-4455-9c37-b10b499e5b8d\") " Nov 03 22:25:26.862146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.862034 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-xks-ca\") pod \"bc17242f-1ea8-421a-8fc8-c6f784447536\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " Nov 03 22:25:26.862146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.862068 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc17242f-1ea8-421a-8fc8-c6f784447536-kube-api-access-55kgc\" (UniqueName: \"kubernetes.io/projected/bc17242f-1ea8-421a-8fc8-c6f784447536-kube-api-access-55kgc\") pod \"bc17242f-1ea8-421a-8fc8-c6f784447536\" (UID: \"bc17242f-1ea8-421a-8fc8-c6f784447536\") " Nov 03 22:25:26.862618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.862370 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-openshift-ca" pod "bc17242f-1ea8-421a-8fc8-c6f784447536" (UID: "bc17242f-1ea8-421a-8fc8-c6f784447536"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:26.862618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.862470 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-openshift-ca" pod "6be1f7db-bf6a-4455-9c37-b10b499e5b8d" (UID: "6be1f7db-bf6a-4455-9c37-b10b499e5b8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:26.862976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.862948 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:26.862976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.862978 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:26.864275 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.864229 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-xks-ca" pod "6be1f7db-bf6a-4455-9c37-b10b499e5b8d" (UID: "6be1f7db-bf6a-4455-9c37-b10b499e5b8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:26.864381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.864366 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc17242f-1ea8-421a-8fc8-c6f784447536-kube-api-access-55kgc" pod "bc17242f-1ea8-421a-8fc8-c6f784447536" (UID: "bc17242f-1ea8-421a-8fc8-c6f784447536"). InnerVolumeSpecName "kube-api-access-55kgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:26.864455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.864423 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-kube-api-access-6d4tw" pod "6be1f7db-bf6a-4455-9c37-b10b499e5b8d" (UID: "6be1f7db-bf6a-4455-9c37-b10b499e5b8d"). InnerVolumeSpecName "kube-api-access-6d4tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:26.865453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.865425 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-xks-ca" pod "bc17242f-1ea8-421a-8fc8-c6f784447536" (UID: "bc17242f-1ea8-421a-8fc8-c6f784447536"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:26.964393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.964361 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:26.964393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.964393 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d4tw\" (UniqueName: \"kubernetes.io/projected/6be1f7db-bf6a-4455-9c37-b10b499e5b8d-kube-api-access-6d4tw\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:26.964615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.964410 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc17242f-1ea8-421a-8fc8-c6f784447536-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:26.964615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:26.964423 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55kgc\" (UniqueName: \"kubernetes.io/projected/bc17242f-1ea8-421a-8fc8-c6f784447536-kube-api-access-55kgc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:27.657674 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.657638 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" Nov 03 22:25:27.657874 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.657675 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl" event={"ID":"bc17242f-1ea8-421a-8fc8-c6f784447536","Type":"ContainerDied","Data":"e84c8946307fde40c0339300620beb1ce95cfa7f15fa3898e599e9c059f363c0"} Nov 03 22:25:27.657874 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.657733 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e84c8946307fde40c0339300620beb1ce95cfa7f15fa3898e599e9c059f363c0" Nov 03 22:25:27.659359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.659335 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" event={"ID":"6be1f7db-bf6a-4455-9c37-b10b499e5b8d","Type":"ContainerDied","Data":"e8740ae3db249f983dba9e150b2621b392e8f57b1d15aa3a7759843a5ad18755"} Nov 03 22:25:27.659359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.659358 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2" Nov 03 22:25:27.659569 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.659371 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8740ae3db249f983dba9e150b2621b392e8f57b1d15aa3a7759843a5ad18755" Nov 03 22:25:27.765794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.765762 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc"] Nov 03 22:25:27.766339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.766325 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bc17242f-1ea8-421a-8fc8-c6f784447536" containerName="revoke-keys" Nov 03 22:25:27.766389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.766341 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc17242f-1ea8-421a-8fc8-c6f784447536" containerName="revoke-keys" Nov 03 22:25:27.766389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.766356 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6be1f7db-bf6a-4455-9c37-b10b499e5b8d" containerName="revoke-keys" Nov 03 22:25:27.766389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.766362 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be1f7db-bf6a-4455-9c37-b10b499e5b8d" containerName="revoke-keys" Nov 03 22:25:27.766493 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.766454 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc17242f-1ea8-421a-8fc8-c6f784447536" containerName="revoke-keys" Nov 03 22:25:27.766493 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.766468 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be1f7db-bf6a-4455-9c37-b10b499e5b8d" containerName="revoke-keys" Nov 03 22:25:27.770220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.770199 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:27.784566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.784545 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:25:27.784749 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.784577 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:25:27.787828 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.787808 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc"] Nov 03 22:25:27.793658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.793640 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:25:27.793783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.793675 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:25:27.875983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.875944 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24kv6\" (UniqueName: \"kubernetes.io/projected/55f3f75d-be17-4f2a-ba55-1f70316410af-kube-api-access-24kv6\") pod \"maas-api-revoke-sub-4fbf6e63-644lc\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:27.876434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.876078 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fbf6e63-644lc\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:27.876434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.876113 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fbf6e63-644lc\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:27.977551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.977462 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fbf6e63-644lc\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:27.977690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.977642 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24kv6\" (UniqueName: \"kubernetes.io/projected/55f3f75d-be17-4f2a-ba55-1f70316410af-kube-api-access-24kv6\") pod \"maas-api-revoke-sub-4fbf6e63-644lc\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:27.977786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.977769 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fbf6e63-644lc\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:27.978451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.978425 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fbf6e63-644lc\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:27.980629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.980603 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fbf6e63-644lc\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:27.998625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:27.998604 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24kv6\" (UniqueName: \"kubernetes.io/projected/55f3f75d-be17-4f2a-ba55-1f70316410af-kube-api-access-24kv6\") pod \"maas-api-revoke-sub-4fbf6e63-644lc\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:28.084693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.084660 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:28.249098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.249023 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8"] Nov 03 22:25:28.254191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.254165 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.271319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.271279 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8"] Nov 03 22:25:28.281390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.281360 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xmcr\" (UniqueName: \"kubernetes.io/projected/0e55a11b-c674-49f4-9090-caedaa7dff18-kube-api-access-7xmcr\") pod \"maas-api-revoke-sub-a7ef02f9-6f6m8\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.281390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.281394 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ef02f9-6f6m8\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.281591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.281511 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ef02f9-6f6m8\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.383050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.383025 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ef02f9-6f6m8\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.383170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.383157 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xmcr\" (UniqueName: \"kubernetes.io/projected/0e55a11b-c674-49f4-9090-caedaa7dff18-kube-api-access-7xmcr\") pod \"maas-api-revoke-sub-a7ef02f9-6f6m8\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.383212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.383184 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ef02f9-6f6m8\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.383901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.383873 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ef02f9-6f6m8\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.384428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.384409 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc"] Nov 03 22:25:28.384944 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:28.384913 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f3f75d_be17_4f2a_ba55_1f70316410af.slice/crio-5df78f2b9aa2769b11e0192455a0e3e202508849f6b2bf576c874de047c29a63 WatchSource:0}: Error finding container 5df78f2b9aa2769b11e0192455a0e3e202508849f6b2bf576c874de047c29a63: Status 404 returned error can't find the container with id 5df78f2b9aa2769b11e0192455a0e3e202508849f6b2bf576c874de047c29a63 Nov 03 22:25:28.385540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.385481 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ef02f9-6f6m8\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.404592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.404566 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xmcr\" (UniqueName: \"kubernetes.io/projected/0e55a11b-c674-49f4-9090-caedaa7dff18-kube-api-access-7xmcr\") pod \"maas-api-revoke-sub-a7ef02f9-6f6m8\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.568228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.568200 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:28.669497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.669462 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" event={"ID":"55f3f75d-be17-4f2a-ba55-1f70316410af","Type":"ContainerStarted","Data":"b4602f14160cf569e2886d91b5d80b9eab7d1261936b027bd6b065fb2ea52857"} Nov 03 22:25:28.669702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.669524 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" event={"ID":"55f3f75d-be17-4f2a-ba55-1f70316410af","Type":"ContainerStarted","Data":"5df78f2b9aa2769b11e0192455a0e3e202508849f6b2bf576c874de047c29a63"} Nov 03 22:25:28.708409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.708365 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" podStartSLOduration=1.708351406 podStartE2EDuration="1.708351406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:28.704809664 +0000 UTC m=+755.108750627" watchObservedRunningTime="2026-11-03 22:25:28.708351406 +0000 UTC m=+755.112292380" Nov 03 22:25:28.867174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:28.867143 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8"] Nov 03 22:25:28.867864 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:28.867835 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e55a11b_c674_49f4_9090_caedaa7dff18.slice/crio-8c374bc3c44cc2e9399f917dd034b029cea038d668939136086e6182110191f9 WatchSource:0}: Error finding container 8c374bc3c44cc2e9399f917dd034b029cea038d668939136086e6182110191f9: Status 404 returned error can't find the container with id 8c374bc3c44cc2e9399f917dd034b029cea038d668939136086e6182110191f9 Nov 03 22:25:29.676952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:29.676918 4903 generic.go:356] "Generic (PLEG): container finished" podID="55f3f75d-be17-4f2a-ba55-1f70316410af" containerID="b4602f14160cf569e2886d91b5d80b9eab7d1261936b027bd6b065fb2ea52857" exitCode=0 Nov 03 22:25:29.677382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:29.677013 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" event={"ID":"55f3f75d-be17-4f2a-ba55-1f70316410af","Type":"ContainerDied","Data":"b4602f14160cf569e2886d91b5d80b9eab7d1261936b027bd6b065fb2ea52857"} Nov 03 22:25:29.678854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:29.678825 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" event={"ID":"0e55a11b-c674-49f4-9090-caedaa7dff18","Type":"ContainerStarted","Data":"41fb44baa609bf5b9c05b86c128016568b0bd6055373d429ec0dcf9304d253fa"} Nov 03 22:25:29.678983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:29.678874 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" event={"ID":"0e55a11b-c674-49f4-9090-caedaa7dff18","Type":"ContainerStarted","Data":"8c374bc3c44cc2e9399f917dd034b029cea038d668939136086e6182110191f9"} Nov 03 22:25:29.775586 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:29.775541 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" podStartSLOduration=1.775527699 podStartE2EDuration="1.775527699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:29.773570669 +0000 UTC m=+756.177511645" watchObservedRunningTime="2026-11-03 22:25:29.775527699 +0000 UTC m=+756.179468673" Nov 03 22:25:30.687667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:30.687631 4903 generic.go:356] "Generic (PLEG): container finished" podID="0e55a11b-c674-49f4-9090-caedaa7dff18" containerID="41fb44baa609bf5b9c05b86c128016568b0bd6055373d429ec0dcf9304d253fa" exitCode=0 Nov 03 22:25:30.688062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:30.687719 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" event={"ID":"0e55a11b-c674-49f4-9090-caedaa7dff18","Type":"ContainerDied","Data":"41fb44baa609bf5b9c05b86c128016568b0bd6055373d429ec0dcf9304d253fa"} Nov 03 22:25:30.817687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:30.817658 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:25:30.883771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:30.883747 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:31.012153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.012065 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-xks-ca\") pod \"55f3f75d-be17-4f2a-ba55-1f70316410af\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " Nov 03 22:25:31.012359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.012181 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f3f75d-be17-4f2a-ba55-1f70316410af-kube-api-access-24kv6\" (UniqueName: \"kubernetes.io/projected/55f3f75d-be17-4f2a-ba55-1f70316410af-kube-api-access-24kv6\") pod \"55f3f75d-be17-4f2a-ba55-1f70316410af\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " Nov 03 22:25:31.012359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.012231 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-openshift-ca\") pod \"55f3f75d-be17-4f2a-ba55-1f70316410af\" (UID: \"55f3f75d-be17-4f2a-ba55-1f70316410af\") " Nov 03 22:25:31.012629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.012605 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-openshift-ca" pod "55f3f75d-be17-4f2a-ba55-1f70316410af" (UID: "55f3f75d-be17-4f2a-ba55-1f70316410af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:31.014170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.014143 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-xks-ca" pod "55f3f75d-be17-4f2a-ba55-1f70316410af" (UID: "55f3f75d-be17-4f2a-ba55-1f70316410af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:31.014287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.014260 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f3f75d-be17-4f2a-ba55-1f70316410af-kube-api-access-24kv6" pod "55f3f75d-be17-4f2a-ba55-1f70316410af" (UID: "55f3f75d-be17-4f2a-ba55-1f70316410af"). InnerVolumeSpecName "kube-api-access-24kv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:31.113698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.113667 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:31.113698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.113692 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24kv6\" (UniqueName: \"kubernetes.io/projected/55f3f75d-be17-4f2a-ba55-1f70316410af-kube-api-access-24kv6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:31.113698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.113702 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f75d-be17-4f2a-ba55-1f70316410af-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:31.144140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.144105 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh"] Nov 03 22:25:31.147335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.144819 4903 cpu_manager.go:414] "containerMap: removing container" podUID="55f3f75d-be17-4f2a-ba55-1f70316410af" containerName="revoke-keys" Nov 03 22:25:31.147335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.144841 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f3f75d-be17-4f2a-ba55-1f70316410af" containerName="revoke-keys" Nov 03 22:25:31.147335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.145032 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f3f75d-be17-4f2a-ba55-1f70316410af" containerName="revoke-keys" Nov 03 22:25:31.148989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.148964 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.164143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.164092 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e19850\"/\"distinct-w1-e19850-kserve-self-signed-certs\"" Nov 03 22:25:31.168262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.168237 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh"] Nov 03 22:25:31.214693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.214660 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tmp-dir\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.214836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.214787 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gg9r\" (UniqueName: \"kubernetes.io/projected/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-kube-api-access-5gg9r\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.214836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.214826 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tls-certs\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.214957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.214876 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-model-cache\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.215029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.215009 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-dshm\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.215079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.215066 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-home\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.316454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.316379 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tls-certs\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.316454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.316449 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-model-cache\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.316634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.316501 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-dshm\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.316634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.316560 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-home\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.316634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.316603 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tmp-dir\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.316736 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.316683 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gg9r\" (UniqueName: \"kubernetes.io/projected/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-kube-api-access-5gg9r\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.316985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.316950 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-model-cache\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.316985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.316968 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-home\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.317130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.317019 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tmp-dir\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.318843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.318827 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-dshm\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.319110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.319094 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tls-certs\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.337665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.337625 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gg9r\" (UniqueName: \"kubernetes.io/projected/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-kube-api-access-5gg9r\") pod \"distinct-w1-e19850-kserve-c6f6cc95d-4r6dh\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.464096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.464062 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:31.713392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.711274 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" Nov 03 22:25:31.713392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.711316 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc" event={"ID":"55f3f75d-be17-4f2a-ba55-1f70316410af","Type":"ContainerDied","Data":"5df78f2b9aa2769b11e0192455a0e3e202508849f6b2bf576c874de047c29a63"} Nov 03 22:25:31.713392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.711411 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5df78f2b9aa2769b11e0192455a0e3e202508849f6b2bf576c874de047c29a63" Nov 03 22:25:31.803985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:31.803949 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh"] Nov 03 22:25:31.808719 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:31.808686 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbd8a10b_f10c_4d22_8d3d_d3ed78acbdef.slice/crio-325846d298215a26688444b8b3493a037b24b79d9f74ad4c012d08b56b895342 WatchSource:0}: Error finding container 325846d298215a26688444b8b3493a037b24b79d9f74ad4c012d08b56b895342: Status 404 returned error can't find the container with id 325846d298215a26688444b8b3493a037b24b79d9f74ad4c012d08b56b895342 Nov 03 22:25:32.046985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.046944 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:32.189204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.185628 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-openshift-ca\") pod \"0e55a11b-c674-49f4-9090-caedaa7dff18\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " Nov 03 22:25:32.189204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.185805 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e55a11b-c674-49f4-9090-caedaa7dff18-kube-api-access-7xmcr\" (UniqueName: \"kubernetes.io/projected/0e55a11b-c674-49f4-9090-caedaa7dff18-kube-api-access-7xmcr\") pod \"0e55a11b-c674-49f4-9090-caedaa7dff18\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " Nov 03 22:25:32.189204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.185934 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-xks-ca\") pod \"0e55a11b-c674-49f4-9090-caedaa7dff18\" (UID: \"0e55a11b-c674-49f4-9090-caedaa7dff18\") " Nov 03 22:25:32.206480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.199624 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-openshift-ca" pod "0e55a11b-c674-49f4-9090-caedaa7dff18" (UID: "0e55a11b-c674-49f4-9090-caedaa7dff18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:32.214836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.214760 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-xks-ca" pod "0e55a11b-c674-49f4-9090-caedaa7dff18" (UID: "0e55a11b-c674-49f4-9090-caedaa7dff18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:32.218995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.218903 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e55a11b-c674-49f4-9090-caedaa7dff18-kube-api-access-7xmcr" pod "0e55a11b-c674-49f4-9090-caedaa7dff18" (UID: "0e55a11b-c674-49f4-9090-caedaa7dff18"). InnerVolumeSpecName "kube-api-access-7xmcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:32.294316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.294124 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xmcr\" (UniqueName: \"kubernetes.io/projected/0e55a11b-c674-49f4-9090-caedaa7dff18-kube-api-access-7xmcr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:32.294316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.294168 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:32.294316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.294185 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e55a11b-c674-49f4-9090-caedaa7dff18-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:32.719979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.719949 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" Nov 03 22:25:32.720475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.719952 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8" event={"ID":"0e55a11b-c674-49f4-9090-caedaa7dff18","Type":"ContainerDied","Data":"8c374bc3c44cc2e9399f917dd034b029cea038d668939136086e6182110191f9"} Nov 03 22:25:32.720475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.720085 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c374bc3c44cc2e9399f917dd034b029cea038d668939136086e6182110191f9" Nov 03 22:25:32.727311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.723984 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" event={"ID":"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef","Type":"ContainerStarted","Data":"905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153"} Nov 03 22:25:32.727311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.724037 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" event={"ID":"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef","Type":"ContainerStarted","Data":"325846d298215a26688444b8b3493a037b24b79d9f74ad4c012d08b56b895342"} Nov 03 22:25:32.765555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:32.765443 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podStartSLOduration=1.7654251090000002 podStartE2EDuration="1.765425109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:32.760562867 +0000 UTC m=+759.164503843" watchObservedRunningTime="2026-11-03 22:25:32.765425109 +0000 UTC m=+759.169366086" Nov 03 22:25:33.466323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:33.464384 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:36.832270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.832177 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6"] Nov 03 22:25:36.833177 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.833037 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0e55a11b-c674-49f4-9090-caedaa7dff18" containerName="revoke-keys" Nov 03 22:25:36.833386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.833358 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e55a11b-c674-49f4-9090-caedaa7dff18" containerName="revoke-keys" Nov 03 22:25:36.833544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.833526 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e55a11b-c674-49f4-9090-caedaa7dff18" containerName="revoke-keys" Nov 03 22:25:36.837282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.837261 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:36.837480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.837458 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn"] Nov 03 22:25:36.841866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.841848 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.856657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.853573 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:25:36.856657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.853609 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:25:36.856971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.856945 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:25:36.857520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.857497 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92682aab-47zwn\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.857520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.857556 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92682aab-47zwn\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.857520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.857649 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eff0233e-45pc6\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:36.857928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.857715 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrr2d\" (UniqueName: \"kubernetes.io/projected/c36caeba-0f0a-4345-bbcb-96b6cfd57913-kube-api-access-wrr2d\") pod \"maas-api-revoke-sub-eff0233e-45pc6\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:36.857928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.857768 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eff0233e-45pc6\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:36.857928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.857859 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6xm\" (UniqueName: \"kubernetes.io/projected/980313da-60b3-45e8-935e-96cfd1f77812-kube-api-access-dq6xm\") pod \"maas-api-revoke-sub-92682aab-47zwn\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.861317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.861279 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn"] Nov 03 22:25:36.864064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.864041 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:25:36.868716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.868692 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6"] Nov 03 22:25:36.958916 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.958871 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eff0233e-45pc6\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:36.959070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.958930 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrr2d\" (UniqueName: \"kubernetes.io/projected/c36caeba-0f0a-4345-bbcb-96b6cfd57913-kube-api-access-wrr2d\") pod \"maas-api-revoke-sub-eff0233e-45pc6\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:36.959070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.958963 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eff0233e-45pc6\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:36.959070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.959017 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6xm\" (UniqueName: \"kubernetes.io/projected/980313da-60b3-45e8-935e-96cfd1f77812-kube-api-access-dq6xm\") pod \"maas-api-revoke-sub-92682aab-47zwn\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.959070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.959065 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92682aab-47zwn\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.959287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.959102 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92682aab-47zwn\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.959928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.959899 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92682aab-47zwn\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.960043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.959899 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eff0233e-45pc6\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:36.961550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.961527 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eff0233e-45pc6\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:36.961626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.961526 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92682aab-47zwn\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.980116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.980093 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6xm\" (UniqueName: \"kubernetes.io/projected/980313da-60b3-45e8-935e-96cfd1f77812-kube-api-access-dq6xm\") pod \"maas-api-revoke-sub-92682aab-47zwn\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:36.980228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:36.980203 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrr2d\" (UniqueName: \"kubernetes.io/projected/c36caeba-0f0a-4345-bbcb-96b6cfd57913-kube-api-access-wrr2d\") pod \"maas-api-revoke-sub-eff0233e-45pc6\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:37.154106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.154074 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4"] Nov 03 22:25:37.157139 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.157114 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:37.161637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.161480 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.166819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.166799 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:37.184537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.182802 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4"] Nov 03 22:25:37.261925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.261896 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d1afc4-dzrx4\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.262084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.261933 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d1afc4-dzrx4\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.262141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.262079 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95fph\" (UniqueName: \"kubernetes.io/projected/f8719195-685c-4781-908b-58a9d3292b8c-kube-api-access-95fph\") pod \"maas-api-revoke-sub-c4d1afc4-dzrx4\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.363622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.363593 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d1afc4-dzrx4\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.363622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.363628 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d1afc4-dzrx4\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.363829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.363679 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95fph\" (UniqueName: \"kubernetes.io/projected/f8719195-685c-4781-908b-58a9d3292b8c-kube-api-access-95fph\") pod \"maas-api-revoke-sub-c4d1afc4-dzrx4\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.364366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.364344 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d1afc4-dzrx4\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.365960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.365938 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d1afc4-dzrx4\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.384818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.384798 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95fph\" (UniqueName: \"kubernetes.io/projected/f8719195-685c-4781-908b-58a9d3292b8c-kube-api-access-95fph\") pod \"maas-api-revoke-sub-c4d1afc4-dzrx4\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.485405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.485379 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6"] Nov 03 22:25:37.486654 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:37.486629 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36caeba_0f0a_4345_bbcb_96b6cfd57913.slice/crio-3756112f2b5e4dad9a1b0e0ec1f95d9cba8b0906e505c3a616af457202d1e84c WatchSource:0}: Error finding container 3756112f2b5e4dad9a1b0e0ec1f95d9cba8b0906e505c3a616af457202d1e84c: Status 404 returned error can't find the container with id 3756112f2b5e4dad9a1b0e0ec1f95d9cba8b0906e505c3a616af457202d1e84c Nov 03 22:25:37.495648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.495624 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn"] Nov 03 22:25:37.498724 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:37.498691 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980313da_60b3_45e8_935e_96cfd1f77812.slice/crio-0b94ed696910e9292c44dc5a9e39fb2bb0b34d5dfbfe8d1e9831720c9b8555d7 WatchSource:0}: Error finding container 0b94ed696910e9292c44dc5a9e39fb2bb0b34d5dfbfe8d1e9831720c9b8555d7: Status 404 returned error can't find the container with id 0b94ed696910e9292c44dc5a9e39fb2bb0b34d5dfbfe8d1e9831720c9b8555d7 Nov 03 22:25:37.520190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.520169 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:37.763411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.763324 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" event={"ID":"980313da-60b3-45e8-935e-96cfd1f77812","Type":"ContainerStarted","Data":"0b94ed696910e9292c44dc5a9e39fb2bb0b34d5dfbfe8d1e9831720c9b8555d7"} Nov 03 22:25:37.765434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.765404 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" event={"ID":"c36caeba-0f0a-4345-bbcb-96b6cfd57913","Type":"ContainerStarted","Data":"3756112f2b5e4dad9a1b0e0ec1f95d9cba8b0906e505c3a616af457202d1e84c"} Nov 03 22:25:37.824076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:37.824049 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4"] Nov 03 22:25:37.825771 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:37.825739 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8719195_685c_4781_908b_58a9d3292b8c.slice/crio-5133b638ffea244b1aeaef354ebd3dc18d80a340954bccb8b032cdd47cef179b WatchSource:0}: Error finding container 5133b638ffea244b1aeaef354ebd3dc18d80a340954bccb8b032cdd47cef179b: Status 404 returned error can't find the container with id 5133b638ffea244b1aeaef354ebd3dc18d80a340954bccb8b032cdd47cef179b Nov 03 22:25:38.773572 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:38.773489 4903 generic.go:356] "Generic (PLEG): container finished" podID="980313da-60b3-45e8-935e-96cfd1f77812" containerID="5dd3f77f6c8d6d317ae6ee2a9964947b8909bfa06736ae089616b29ff34fb821" exitCode=0 Nov 03 22:25:38.773572 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:38.773529 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" event={"ID":"980313da-60b3-45e8-935e-96cfd1f77812","Type":"ContainerDied","Data":"5dd3f77f6c8d6d317ae6ee2a9964947b8909bfa06736ae089616b29ff34fb821"} Nov 03 22:25:38.775206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:38.775183 4903 generic.go:356] "Generic (PLEG): container finished" podID="c36caeba-0f0a-4345-bbcb-96b6cfd57913" containerID="c27b41d7aabaf7f6648391e7ba35373eedf15bcba6d1544b16cc94185b927060" exitCode=0 Nov 03 22:25:38.775352 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:38.775254 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" event={"ID":"c36caeba-0f0a-4345-bbcb-96b6cfd57913","Type":"ContainerDied","Data":"c27b41d7aabaf7f6648391e7ba35373eedf15bcba6d1544b16cc94185b927060"} Nov 03 22:25:38.776688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:38.776668 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" event={"ID":"f8719195-685c-4781-908b-58a9d3292b8c","Type":"ContainerStarted","Data":"9ab9eaa7883482b4bf46a9af5a88aeee7e8d1994cadc2e51e2fe68e923edca7b"} Nov 03 22:25:38.776759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:38.776709 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" event={"ID":"f8719195-685c-4781-908b-58a9d3292b8c","Type":"ContainerStarted","Data":"5133b638ffea244b1aeaef354ebd3dc18d80a340954bccb8b032cdd47cef179b"} Nov 03 22:25:38.846020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:38.845970 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" podStartSLOduration=1.845954239 podStartE2EDuration="1.845954239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:38.843715179 +0000 UTC m=+765.247656145" watchObservedRunningTime="2026-11-03 22:25:38.845954239 +0000 UTC m=+765.249895215" Nov 03 22:25:39.785408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:39.785371 4903 generic.go:356] "Generic (PLEG): container finished" podID="f8719195-685c-4781-908b-58a9d3292b8c" containerID="9ab9eaa7883482b4bf46a9af5a88aeee7e8d1994cadc2e51e2fe68e923edca7b" exitCode=0 Nov 03 22:25:39.785856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:39.785442 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" event={"ID":"f8719195-685c-4781-908b-58a9d3292b8c","Type":"ContainerDied","Data":"9ab9eaa7883482b4bf46a9af5a88aeee7e8d1994cadc2e51e2fe68e923edca7b"} Nov 03 22:25:39.992825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:39.992800 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:39.996173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:39.996155 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:40.094391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.094283 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-openshift-ca\") pod \"980313da-60b3-45e8-935e-96cfd1f77812\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " Nov 03 22:25:40.094555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.094386 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-xks-ca\") pod \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " Nov 03 22:25:40.094555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.094512 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980313da-60b3-45e8-935e-96cfd1f77812-kube-api-access-dq6xm\" (UniqueName: \"kubernetes.io/projected/980313da-60b3-45e8-935e-96cfd1f77812-kube-api-access-dq6xm\") pod \"980313da-60b3-45e8-935e-96cfd1f77812\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " Nov 03 22:25:40.094555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.094541 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36caeba-0f0a-4345-bbcb-96b6cfd57913-kube-api-access-wrr2d\" (UniqueName: \"kubernetes.io/projected/c36caeba-0f0a-4345-bbcb-96b6cfd57913-kube-api-access-wrr2d\") pod \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " Nov 03 22:25:40.094726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.094599 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-xks-ca\") pod \"980313da-60b3-45e8-935e-96cfd1f77812\" (UID: \"980313da-60b3-45e8-935e-96cfd1f77812\") " Nov 03 22:25:40.094726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.094647 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-openshift-ca\") pod \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\" (UID: \"c36caeba-0f0a-4345-bbcb-96b6cfd57913\") " Nov 03 22:25:40.094726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.094667 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-openshift-ca" pod "980313da-60b3-45e8-935e-96cfd1f77812" (UID: "980313da-60b3-45e8-935e-96cfd1f77812"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:40.095121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.095096 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:40.095250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.095153 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-openshift-ca" pod "c36caeba-0f0a-4345-bbcb-96b6cfd57913" (UID: "c36caeba-0f0a-4345-bbcb-96b6cfd57913"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:40.096829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.096800 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36caeba-0f0a-4345-bbcb-96b6cfd57913-kube-api-access-wrr2d" pod "c36caeba-0f0a-4345-bbcb-96b6cfd57913" (UID: "c36caeba-0f0a-4345-bbcb-96b6cfd57913"). InnerVolumeSpecName "kube-api-access-wrr2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:40.096934 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.096839 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-xks-ca" pod "c36caeba-0f0a-4345-bbcb-96b6cfd57913" (UID: "c36caeba-0f0a-4345-bbcb-96b6cfd57913"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:40.097043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.097022 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980313da-60b3-45e8-935e-96cfd1f77812-kube-api-access-dq6xm" pod "980313da-60b3-45e8-935e-96cfd1f77812" (UID: "980313da-60b3-45e8-935e-96cfd1f77812"). InnerVolumeSpecName "kube-api-access-dq6xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:40.097218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.097201 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-xks-ca" pod "980313da-60b3-45e8-935e-96cfd1f77812" (UID: "980313da-60b3-45e8-935e-96cfd1f77812"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:40.195807 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.195775 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6xm\" (UniqueName: \"kubernetes.io/projected/980313da-60b3-45e8-935e-96cfd1f77812-kube-api-access-dq6xm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:40.195807 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.195804 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrr2d\" (UniqueName: \"kubernetes.io/projected/c36caeba-0f0a-4345-bbcb-96b6cfd57913-kube-api-access-wrr2d\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:40.195807 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.195814 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980313da-60b3-45e8-935e-96cfd1f77812-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:40.196005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.195825 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:40.196005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.195834 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c36caeba-0f0a-4345-bbcb-96b6cfd57913-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:40.792790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.792758 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" Nov 03 22:25:40.793176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.792799 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn" event={"ID":"980313da-60b3-45e8-935e-96cfd1f77812","Type":"ContainerDied","Data":"0b94ed696910e9292c44dc5a9e39fb2bb0b34d5dfbfe8d1e9831720c9b8555d7"} Nov 03 22:25:40.793176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.792848 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b94ed696910e9292c44dc5a9e39fb2bb0b34d5dfbfe8d1e9831720c9b8555d7" Nov 03 22:25:40.794267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.794238 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" Nov 03 22:25:40.794414 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.794267 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6" event={"ID":"c36caeba-0f0a-4345-bbcb-96b6cfd57913","Type":"ContainerDied","Data":"3756112f2b5e4dad9a1b0e0ec1f95d9cba8b0906e505c3a616af457202d1e84c"} Nov 03 22:25:40.794414 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.794325 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3756112f2b5e4dad9a1b0e0ec1f95d9cba8b0906e505c3a616af457202d1e84c" Nov 03 22:25:40.943826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:40.943804 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:41.004513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.004485 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-openshift-ca\") pod \"f8719195-685c-4781-908b-58a9d3292b8c\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " Nov 03 22:25:41.004683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.004588 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8719195-685c-4781-908b-58a9d3292b8c-kube-api-access-95fph\" (UniqueName: \"kubernetes.io/projected/f8719195-685c-4781-908b-58a9d3292b8c-kube-api-access-95fph\") pod \"f8719195-685c-4781-908b-58a9d3292b8c\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " Nov 03 22:25:41.004683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.004638 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-xks-ca\") pod \"f8719195-685c-4781-908b-58a9d3292b8c\" (UID: \"f8719195-685c-4781-908b-58a9d3292b8c\") " Nov 03 22:25:41.004906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.004879 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-openshift-ca" pod "f8719195-685c-4781-908b-58a9d3292b8c" (UID: "f8719195-685c-4781-908b-58a9d3292b8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:41.005161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.005141 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:41.006691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.006665 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8719195-685c-4781-908b-58a9d3292b8c-kube-api-access-95fph" pod "f8719195-685c-4781-908b-58a9d3292b8c" (UID: "f8719195-685c-4781-908b-58a9d3292b8c"). InnerVolumeSpecName "kube-api-access-95fph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:41.007083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.007054 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-xks-ca" pod "f8719195-685c-4781-908b-58a9d3292b8c" (UID: "f8719195-685c-4781-908b-58a9d3292b8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:41.106472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.106385 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95fph\" (UniqueName: \"kubernetes.io/projected/f8719195-685c-4781-908b-58a9d3292b8c-kube-api-access-95fph\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:41.106472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.106419 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8719195-685c-4781-908b-58a9d3292b8c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:41.464724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.464690 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:41.478256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.478232 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:41.802018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.801989 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" Nov 03 22:25:41.802429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.802020 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4" event={"ID":"f8719195-685c-4781-908b-58a9d3292b8c","Type":"ContainerDied","Data":"5133b638ffea244b1aeaef354ebd3dc18d80a340954bccb8b032cdd47cef179b"} Nov 03 22:25:41.802429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.802086 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5133b638ffea244b1aeaef354ebd3dc18d80a340954bccb8b032cdd47cef179b" Nov 03 22:25:41.813546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:41.813522 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:25:42.759460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.759419 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd"] Nov 03 22:25:42.760021 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.760001 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c36caeba-0f0a-4345-bbcb-96b6cfd57913" containerName="revoke-keys" Nov 03 22:25:42.760021 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.760019 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36caeba-0f0a-4345-bbcb-96b6cfd57913" containerName="revoke-keys" Nov 03 22:25:42.760202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.760047 4903 cpu_manager.go:414] "containerMap: removing container" podUID="980313da-60b3-45e8-935e-96cfd1f77812" containerName="revoke-keys" Nov 03 22:25:42.760202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.760053 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980313da-60b3-45e8-935e-96cfd1f77812" containerName="revoke-keys" Nov 03 22:25:42.760202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.760076 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f8719195-685c-4781-908b-58a9d3292b8c" containerName="revoke-keys" Nov 03 22:25:42.760202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.760081 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8719195-685c-4781-908b-58a9d3292b8c" containerName="revoke-keys" Nov 03 22:25:42.760202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.760183 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8719195-685c-4781-908b-58a9d3292b8c" containerName="revoke-keys" Nov 03 22:25:42.760202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.760194 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36caeba-0f0a-4345-bbcb-96b6cfd57913" containerName="revoke-keys" Nov 03 22:25:42.760202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.760205 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="980313da-60b3-45e8-935e-96cfd1f77812" containerName="revoke-keys" Nov 03 22:25:42.763688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.763671 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.788724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.788686 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-50d738-openshift-default-dockercfg-jhlmb\"" Nov 03 22:25:42.792267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.792241 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd"] Nov 03 22:25:42.823188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.823149 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9h8p\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-kube-api-access-b9h8p\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.828710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.828675 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.828897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.828751 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5da792a-d829-4789-b232-ab527d42ac6e-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.828897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.828792 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.828897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.828849 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5da792a-d829-4789-b232-ab527d42ac6e-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.832245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.828911 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.832245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.828972 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.832245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.829034 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.832245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.829071 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.930620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.930588 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9h8p\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-kube-api-access-b9h8p\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.930824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.930630 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.930824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.930663 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5da792a-d829-4789-b232-ab527d42ac6e-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.930824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.930692 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.930824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.930726 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5da792a-d829-4789-b232-ab527d42ac6e-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.931044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.930870 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.931044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.930940 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.931044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.930998 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.931191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.931064 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.931191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.931130 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.931191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.931067 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.931398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.931376 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.931478 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.931461 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.931786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.931767 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5da792a-d829-4789-b232-ab527d42ac6e-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.933026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.933007 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.933170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.933152 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5da792a-d829-4789-b232-ab527d42ac6e-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.952057 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.952036 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9h8p\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-kube-api-access-b9h8p\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:42.952155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:42.952092 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:43.080741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:43.080655 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:43.398109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:43.398076 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd"] Nov 03 22:25:43.398363 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:43.398342 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5da792a_d829_4789_b232_ab527d42ac6e.slice/crio-4701d77f752bf7bb7f04f68d7548d7a66dfb7f81cb166868fa3c045d2a15dfb5 WatchSource:0}: Error finding container 4701d77f752bf7bb7f04f68d7548d7a66dfb7f81cb166868fa3c045d2a15dfb5: Status 404 returned error can't find the container with id 4701d77f752bf7bb7f04f68d7548d7a66dfb7f81cb166868fa3c045d2a15dfb5 Nov 03 22:25:43.400660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:43.400632 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:25:43.400727 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:43.400692 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:25:43.400727 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:43.400722 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:25:43.817966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:43.817929 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" event={"ID":"d5da792a-d829-4789-b232-ab527d42ac6e","Type":"ContainerStarted","Data":"17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694"} Nov 03 22:25:43.818122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:43.817984 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" event={"ID":"d5da792a-d829-4789-b232-ab527d42ac6e","Type":"ContainerStarted","Data":"4701d77f752bf7bb7f04f68d7548d7a66dfb7f81cb166868fa3c045d2a15dfb5"} Nov 03 22:25:43.864322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:43.862033 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" podStartSLOduration=1.862013272 podStartE2EDuration="1.862013272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:43.860055188 +0000 UTC m=+770.263996164" watchObservedRunningTime="2026-11-03 22:25:43.862013272 +0000 UTC m=+770.265954248" Nov 03 22:25:44.081212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:44.081118 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:44.082746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:44.082717 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:25:44.082876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:44.082773 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:25:45.087316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:45.086727 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:25:45.087316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:45.086796 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:25:46.081948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:46.081915 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:25:46.082133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:46.081976 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:25:47.081033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.080996 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:25:47.081456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.081050 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:25:47.174143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.174103 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5"] Nov 03 22:25:47.178933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.178910 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.194152 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.194125 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5"] Nov 03 22:25:47.200753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.200709 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:25:47.200904 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.200816 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:25:47.200904 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.200833 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:25:47.200904 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.200853 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:25:47.375780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.375692 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q52s\" (UniqueName: \"kubernetes.io/projected/47c6c00e-3801-48bb-853f-72ec8ef5d831-kube-api-access-8q52s\") pod \"maas-api-revoke-sub-8e8d15ed-7lrr5\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.375780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.375755 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e8d15ed-7lrr5\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.375969 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.375870 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e8d15ed-7lrr5\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.476623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.476583 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e8d15ed-7lrr5\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.476797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.476657 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e8d15ed-7lrr5\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.476856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.476800 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q52s\" (UniqueName: \"kubernetes.io/projected/47c6c00e-3801-48bb-853f-72ec8ef5d831-kube-api-access-8q52s\") pod \"maas-api-revoke-sub-8e8d15ed-7lrr5\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.477490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.477455 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e8d15ed-7lrr5\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.479368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.479340 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e8d15ed-7lrr5\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.481334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.481315 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6"] Nov 03 22:25:47.485732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.485714 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.499163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.499136 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q52s\" (UniqueName: \"kubernetes.io/projected/47c6c00e-3801-48bb-853f-72ec8ef5d831-kube-api-access-8q52s\") pod \"maas-api-revoke-sub-8e8d15ed-7lrr5\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.506205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.506181 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6"] Nov 03 22:25:47.577346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.577309 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7kf\" (UniqueName: \"kubernetes.io/projected/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-kube-api-access-mq7kf\") pod \"maas-api-revoke-sub-36f20ab1-l7vg6\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.577346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.577353 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f20ab1-l7vg6\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.577564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.577471 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f20ab1-l7vg6\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.678428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.678394 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7kf\" (UniqueName: \"kubernetes.io/projected/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-kube-api-access-mq7kf\") pod \"maas-api-revoke-sub-36f20ab1-l7vg6\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.678609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.678435 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f20ab1-l7vg6\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.678609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.678474 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f20ab1-l7vg6\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.679201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.679175 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f20ab1-l7vg6\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.680816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.680799 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f20ab1-l7vg6\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.701023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.700995 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7kf\" (UniqueName: \"kubernetes.io/projected/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-kube-api-access-mq7kf\") pod \"maas-api-revoke-sub-36f20ab1-l7vg6\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:47.794852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.794823 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:47.804366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:47.804343 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:48.081561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.081526 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:25:48.081961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.081609 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:25:48.127244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.127211 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5"] Nov 03 22:25:48.135707 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:48.135660 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c6c00e_3801_48bb_853f_72ec8ef5d831.slice/crio-7262b9e8dec5a50f48d9c8f3d96bf7db8da4552045d8f4c104de032e266b6560 WatchSource:0}: Error finding container 7262b9e8dec5a50f48d9c8f3d96bf7db8da4552045d8f4c104de032e266b6560: Status 404 returned error can't find the container with id 7262b9e8dec5a50f48d9c8f3d96bf7db8da4552045d8f4c104de032e266b6560 Nov 03 22:25:48.179709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.178450 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6"] Nov 03 22:25:48.184887 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:25:48.183669 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a73ea1e_ec66_46c5_a37b_aef249c3bc51.slice/crio-f71a1a3b9efe3dbcf7491e55832cfe3b4746a7d2e461c9cd1783fe67b679ef97 WatchSource:0}: Error finding container f71a1a3b9efe3dbcf7491e55832cfe3b4746a7d2e461c9cd1783fe67b679ef97: Status 404 returned error can't find the container with id f71a1a3b9efe3dbcf7491e55832cfe3b4746a7d2e461c9cd1783fe67b679ef97 Nov 03 22:25:48.399191 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:25:48.399157 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:25:48.903789 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.903759 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" event={"ID":"1a73ea1e-ec66-46c5-a37b-aef249c3bc51","Type":"ContainerStarted","Data":"5ed6888275a4df08bd5bb0924adc6f0a5ad255e97177fa1c1d90b8c153534952"} Nov 03 22:25:48.903968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.903820 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" event={"ID":"1a73ea1e-ec66-46c5-a37b-aef249c3bc51","Type":"ContainerStarted","Data":"f71a1a3b9efe3dbcf7491e55832cfe3b4746a7d2e461c9cd1783fe67b679ef97"} Nov 03 22:25:48.905383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.905358 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" event={"ID":"47c6c00e-3801-48bb-853f-72ec8ef5d831","Type":"ContainerStarted","Data":"6112ce8f6c744ab8ba0567a49de34b2914cd548602e1755dba9174c6c2a17014"} Nov 03 22:25:48.905504 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.905397 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" event={"ID":"47c6c00e-3801-48bb-853f-72ec8ef5d831","Type":"ContainerStarted","Data":"7262b9e8dec5a50f48d9c8f3d96bf7db8da4552045d8f4c104de032e266b6560"} Nov 03 22:25:48.939930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.939882 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" podStartSLOduration=1.9398589529999999 podStartE2EDuration="1.939858953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:48.939457525 +0000 UTC m=+775.343398496" watchObservedRunningTime="2026-11-03 22:25:48.939858953 +0000 UTC m=+775.343799928" Nov 03 22:25:48.983580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:48.983530 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" podStartSLOduration=1.983517822 podStartE2EDuration="1.983517822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:25:48.980784377 +0000 UTC m=+775.384725352" watchObservedRunningTime="2026-11-03 22:25:48.983517822 +0000 UTC m=+775.387458797" Nov 03 22:25:49.081770 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:49.081740 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 22:25:49.082130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:49.081795 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 22:25:49.914283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:49.914240 4903 generic.go:356] "Generic (PLEG): container finished" podID="47c6c00e-3801-48bb-853f-72ec8ef5d831" containerID="6112ce8f6c744ab8ba0567a49de34b2914cd548602e1755dba9174c6c2a17014" exitCode=0 Nov 03 22:25:49.914515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:49.914327 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" event={"ID":"47c6c00e-3801-48bb-853f-72ec8ef5d831","Type":"ContainerDied","Data":"6112ce8f6c744ab8ba0567a49de34b2914cd548602e1755dba9174c6c2a17014"} Nov 03 22:25:49.916338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:49.916282 4903 generic.go:356] "Generic (PLEG): container finished" podID="1a73ea1e-ec66-46c5-a37b-aef249c3bc51" containerID="5ed6888275a4df08bd5bb0924adc6f0a5ad255e97177fa1c1d90b8c153534952" exitCode=0 Nov 03 22:25:49.916443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:49.916341 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" event={"ID":"1a73ea1e-ec66-46c5-a37b-aef249c3bc51","Type":"ContainerDied","Data":"5ed6888275a4df08bd5bb0924adc6f0a5ad255e97177fa1c1d90b8c153534952"} Nov 03 22:25:50.775497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:50.775466 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:50.776142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:50.776124 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:50.776886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:50.776866 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:25:51.087465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.087431 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:51.117654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.117627 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-xks-ca\") pod \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " Nov 03 22:25:51.117780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.117666 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-kube-api-access-mq7kf\" (UniqueName: \"kubernetes.io/projected/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-kube-api-access-mq7kf\") pod \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " Nov 03 22:25:51.117780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.117744 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-openshift-ca\") pod \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\" (UID: \"1a73ea1e-ec66-46c5-a37b-aef249c3bc51\") " Nov 03 22:25:51.118343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.118289 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-openshift-ca" pod "1a73ea1e-ec66-46c5-a37b-aef249c3bc51" (UID: "1a73ea1e-ec66-46c5-a37b-aef249c3bc51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:51.120107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.120084 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-xks-ca" pod "1a73ea1e-ec66-46c5-a37b-aef249c3bc51" (UID: "1a73ea1e-ec66-46c5-a37b-aef249c3bc51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:51.120327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.120286 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-kube-api-access-mq7kf" pod "1a73ea1e-ec66-46c5-a37b-aef249c3bc51" (UID: "1a73ea1e-ec66-46c5-a37b-aef249c3bc51"). InnerVolumeSpecName "kube-api-access-mq7kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:51.149346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.149314 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:51.219085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.219044 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-openshift-ca\") pod \"47c6c00e-3801-48bb-853f-72ec8ef5d831\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " Nov 03 22:25:51.219254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.219197 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c6c00e-3801-48bb-853f-72ec8ef5d831-kube-api-access-8q52s\" (UniqueName: \"kubernetes.io/projected/47c6c00e-3801-48bb-853f-72ec8ef5d831-kube-api-access-8q52s\") pod \"47c6c00e-3801-48bb-853f-72ec8ef5d831\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " Nov 03 22:25:51.219254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.219234 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-xks-ca\") pod \"47c6c00e-3801-48bb-853f-72ec8ef5d831\" (UID: \"47c6c00e-3801-48bb-853f-72ec8ef5d831\") " Nov 03 22:25:51.219533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.219505 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-openshift-ca" pod "47c6c00e-3801-48bb-853f-72ec8ef5d831" (UID: "47c6c00e-3801-48bb-853f-72ec8ef5d831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:25:51.219711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.219685 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:51.219711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.219711 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:51.219848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.219726 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7kf\" (UniqueName: \"kubernetes.io/projected/1a73ea1e-ec66-46c5-a37b-aef249c3bc51-kube-api-access-mq7kf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:51.219848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.219741 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:51.221798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.221777 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-xks-ca" pod "47c6c00e-3801-48bb-853f-72ec8ef5d831" (UID: "47c6c00e-3801-48bb-853f-72ec8ef5d831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:25:51.221866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.221799 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c6c00e-3801-48bb-853f-72ec8ef5d831-kube-api-access-8q52s" pod "47c6c00e-3801-48bb-853f-72ec8ef5d831" (UID: "47c6c00e-3801-48bb-853f-72ec8ef5d831"). InnerVolumeSpecName "kube-api-access-8q52s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:25:51.320513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.320442 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q52s\" (UniqueName: \"kubernetes.io/projected/47c6c00e-3801-48bb-853f-72ec8ef5d831-kube-api-access-8q52s\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:51.320513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.320467 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c6c00e-3801-48bb-853f-72ec8ef5d831-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:25:51.930966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.930938 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" Nov 03 22:25:51.931359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.930980 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6" event={"ID":"1a73ea1e-ec66-46c5-a37b-aef249c3bc51","Type":"ContainerDied","Data":"f71a1a3b9efe3dbcf7491e55832cfe3b4746a7d2e461c9cd1783fe67b679ef97"} Nov 03 22:25:51.931359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.931026 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f71a1a3b9efe3dbcf7491e55832cfe3b4746a7d2e461c9cd1783fe67b679ef97" Nov 03 22:25:51.932532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.932508 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" event={"ID":"47c6c00e-3801-48bb-853f-72ec8ef5d831","Type":"ContainerDied","Data":"7262b9e8dec5a50f48d9c8f3d96bf7db8da4552045d8f4c104de032e266b6560"} Nov 03 22:25:51.932661 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.932549 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7262b9e8dec5a50f48d9c8f3d96bf7db8da4552045d8f4c104de032e266b6560" Nov 03 22:25:51.932661 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:51.932553 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5" Nov 03 22:25:57.497522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:25:57.497488 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:26:02.025634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.025598 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq"] Nov 03 22:26:02.026225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.026206 4903 cpu_manager.go:414] "containerMap: removing container" podUID="47c6c00e-3801-48bb-853f-72ec8ef5d831" containerName="revoke-keys" Nov 03 22:26:02.026225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.026224 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c6c00e-3801-48bb-853f-72ec8ef5d831" containerName="revoke-keys" Nov 03 22:26:02.026401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.026282 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1a73ea1e-ec66-46c5-a37b-aef249c3bc51" containerName="revoke-keys" Nov 03 22:26:02.026401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.026303 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a73ea1e-ec66-46c5-a37b-aef249c3bc51" containerName="revoke-keys" Nov 03 22:26:02.026516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.026420 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c6c00e-3801-48bb-853f-72ec8ef5d831" containerName="revoke-keys" Nov 03 22:26:02.026516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.026446 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a73ea1e-ec66-46c5-a37b-aef249c3bc51" containerName="revoke-keys" Nov 03 22:26:02.031683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.031659 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.056481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.056450 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-553753\"/\"distinct-w2-553753-kserve-self-signed-certs\"" Nov 03 22:26:02.061884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.061853 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq"] Nov 03 22:26:02.148240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.148205 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3c564-4549-49be-aba0-e474d0ad07aa-tls-certs\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.148240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.148246 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-model-cache\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.148485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.148283 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-tmp-dir\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.148485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.148472 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-home\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.148579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.148517 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ts8v\" (UniqueName: \"kubernetes.io/projected/d8f3c564-4549-49be-aba0-e474d0ad07aa-kube-api-access-4ts8v\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.148579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.148540 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-dshm\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.249942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.249906 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-home\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.250128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.249956 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ts8v\" (UniqueName: \"kubernetes.io/projected/d8f3c564-4549-49be-aba0-e474d0ad07aa-kube-api-access-4ts8v\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.250128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.249979 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-dshm\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.250128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.250030 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3c564-4549-49be-aba0-e474d0ad07aa-tls-certs\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.250128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.250048 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-model-cache\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.250128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.250072 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-tmp-dir\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.250482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.250411 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-home\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.250482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.250436 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-tmp-dir\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.250584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.250533 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-model-cache\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.252363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.252333 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-dshm\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.252537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.252520 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3c564-4549-49be-aba0-e474d0ad07aa-tls-certs\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.272351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.272317 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ts8v\" (UniqueName: \"kubernetes.io/projected/d8f3c564-4549-49be-aba0-e474d0ad07aa-kube-api-access-4ts8v\") pod \"distinct-w2-553753-kserve-797c58bb75-tf6rq\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.348434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.348347 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:02.658670 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:26:02.658639 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8f3c564_4549_49be_aba0_e474d0ad07aa.slice/crio-9af170a381b40aa12d45cb0b26b9716f6e134e263043ab98111d9c38e598400b WatchSource:0}: Error finding container 9af170a381b40aa12d45cb0b26b9716f6e134e263043ab98111d9c38e598400b: Status 404 returned error can't find the container with id 9af170a381b40aa12d45cb0b26b9716f6e134e263043ab98111d9c38e598400b Nov 03 22:26:02.658972 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:02.658949 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq"] Nov 03 22:26:03.009910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:03.009812 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" event={"ID":"d8f3c564-4549-49be-aba0-e474d0ad07aa","Type":"ContainerStarted","Data":"b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486"} Nov 03 22:26:03.009910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:03.009863 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" event={"ID":"d8f3c564-4549-49be-aba0-e474d0ad07aa","Type":"ContainerStarted","Data":"9af170a381b40aa12d45cb0b26b9716f6e134e263043ab98111d9c38e598400b"} Nov 03 22:26:03.050598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:03.050547 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podStartSLOduration=1.050534478 podStartE2EDuration="1.050534478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:03.048101033 +0000 UTC m=+789.452042007" watchObservedRunningTime="2026-11-03 22:26:03.050534478 +0000 UTC m=+789.454475470" Nov 03 22:26:03.348779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:03.348703 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:12.348790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:12.348750 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:12.364881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:12.364850 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:12.980149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:12.980096 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg"] Nov 03 22:26:12.990468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:12.990439 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.004777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.004745 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg"] Nov 03 22:26:13.098607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.098578 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:26:13.105194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.105163 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.105368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.105204 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.105368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.105277 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b28bb3-915a-4303-a100-84b16be20ebc-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.105470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.105372 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.105470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.105454 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b28bb3-915a-4303-a100-84b16be20ebc-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.105575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.105544 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.105666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.105653 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.105721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.105685 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.105815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.105789 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpsp\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-kube-api-access-pwpsp\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.207577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.207537 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b28bb3-915a-4303-a100-84b16be20ebc-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.207779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.207633 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.207779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.207710 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b28bb3-915a-4303-a100-84b16be20ebc-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.207779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.207747 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.207945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.207837 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.207945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.207863 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.207945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.207904 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpsp\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-kube-api-access-pwpsp\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.208109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.207974 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.208109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.208004 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.208414 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.208387 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b28bb3-915a-4303-a100-84b16be20ebc-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.208582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.208506 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.208689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.208532 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.208788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.208621 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.208788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.208629 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.210699 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.210670 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.210856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.210833 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b28bb3-915a-4303-a100-84b16be20ebc-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.230474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.230401 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpsp\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-kube-api-access-pwpsp\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.230644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.230619 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-rdndg\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.308372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.308320 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:13.633995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.633965 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg"] Nov 03 22:26:13.634580 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:26:13.634556 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b28bb3_915a_4303_a100_84b16be20ebc.slice/crio-23523c692ceba2cc69fd2483032ef95eec732e6c288df6eef69283f9cbf05a07 WatchSource:0}: Error finding container 23523c692ceba2cc69fd2483032ef95eec732e6c288df6eef69283f9cbf05a07: Status 404 returned error can't find the container with id 23523c692ceba2cc69fd2483032ef95eec732e6c288df6eef69283f9cbf05a07 Nov 03 22:26:13.637032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.637005 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:13.637092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.637070 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:13.637130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:13.637097 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:14.093737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:14.093698 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" event={"ID":"00b28bb3-915a-4303-a100-84b16be20ebc","Type":"ContainerStarted","Data":"2054cff75d0acc5785f8e504972c6305b01738c81d67615a4391c12299e0b304"} Nov 03 22:26:14.093955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:14.093761 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" event={"ID":"00b28bb3-915a-4303-a100-84b16be20ebc","Type":"ContainerStarted","Data":"23523c692ceba2cc69fd2483032ef95eec732e6c288df6eef69283f9cbf05a07"} Nov 03 22:26:14.137847 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:14.137789 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" podStartSLOduration=2.137771042 podStartE2EDuration="2.137771042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:14.134081663 +0000 UTC m=+800.538022635" watchObservedRunningTime="2026-11-03 22:26:14.137771042 +0000 UTC m=+800.541712016" Nov 03 22:26:14.309730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:14.309510 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:14.316496 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:14.316454 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-rdndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:26:14.316717 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:14.316529 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:26:15.309176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:15.309142 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-rdndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:26:15.309605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:15.309195 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:26:16.309178 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:16.309145 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-rdndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:26:16.309669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:16.309195 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:26:17.309439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:17.309404 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-rdndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:26:17.309836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:17.309460 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:26:18.308878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:18.308836 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-rdndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:26:18.309081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:18.308896 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:26:18.392925 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:26:18.392891 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:26:19.309647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:19.309616 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-rdndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:26:19.309838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:19.309679 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:26:20.308621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:20.308580 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-rdndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 22:26:20.309106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:20.308649 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 22:26:21.313213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:21.313181 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:21.313721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:21.313643 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:21.314416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:21.314393 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:26:27.030865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.030829 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n"] Nov 03 22:26:27.036814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.036789 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.061636 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.061604 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-434ce3\"/\"distinct-w5-434ce3-kserve-self-signed-certs\"" Nov 03 22:26:27.065250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.065222 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n"] Nov 03 22:26:27.089815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.089778 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-dshm\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.089988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.089860 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltcx\" (UniqueName: \"kubernetes.io/projected/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-kube-api-access-bltcx\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.089988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.089933 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tls-certs\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.090081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.089995 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-home\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.090081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.090064 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-model-cache\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.090191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.090174 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tmp-dir\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.191834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.191801 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-home\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.192037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.191872 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-model-cache\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.192037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.191916 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tmp-dir\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.192037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.191963 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-dshm\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.192037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.192008 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltcx\" (UniqueName: \"kubernetes.io/projected/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-kube-api-access-bltcx\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.192037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.192037 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tls-certs\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.192402 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.192348 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-home\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.192402 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.192377 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tmp-dir\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.192499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.192419 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-model-cache\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.194285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.194258 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-dshm\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.194629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.194613 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tls-certs\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.214084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.214050 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltcx\" (UniqueName: \"kubernetes.io/projected/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-kube-api-access-bltcx\") pod \"distinct-w5-434ce3-kserve-794c49c7dd-dpm7n\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.349932 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.349844 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:27.689105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.689054 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n"] Nov 03 22:26:27.689339 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:26:27.689284 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eb9ea98_1455_4f8e_b439_183fdfb7dc19.slice/crio-f6bb3a1df8be30d727d1b18d2281b47ac22a12302ebda1a7df52036c1e6affb8 WatchSource:0}: Error finding container f6bb3a1df8be30d727d1b18d2281b47ac22a12302ebda1a7df52036c1e6affb8: Status 404 returned error can't find the container with id f6bb3a1df8be30d727d1b18d2281b47ac22a12302ebda1a7df52036c1e6affb8 Nov 03 22:26:27.762464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:27.762433 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:26:28.190367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.190332 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" event={"ID":"9eb9ea98-1455-4f8e-b439-183fdfb7dc19","Type":"ContainerStarted","Data":"39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013"} Nov 03 22:26:28.190758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.190381 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" event={"ID":"9eb9ea98-1455-4f8e-b439-183fdfb7dc19","Type":"ContainerStarted","Data":"f6bb3a1df8be30d727d1b18d2281b47ac22a12302ebda1a7df52036c1e6affb8"} Nov 03 22:26:28.230914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.230859 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podStartSLOduration=1.23084527 podStartE2EDuration="1.23084527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:28.227594757 +0000 UTC m=+814.631535742" watchObservedRunningTime="2026-11-03 22:26:28.23084527 +0000 UTC m=+814.634786245" Nov 03 22:26:28.350313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.350262 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:28.727120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.727085 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6"] Nov 03 22:26:28.731310 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.731280 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.751091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.751064 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-88a4d9-openshift-default-dockercfg-ph54l\"" Nov 03 22:26:28.764871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.764833 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6"] Nov 03 22:26:28.814147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.814109 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.814370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.814182 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db522d78-bd16-46c0-8dbd-f6670cb42c65-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.814370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.814212 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.814513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.814389 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pl48\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-kube-api-access-4pl48\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.814575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.814521 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.814575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.814546 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.814575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.814567 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.814695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.814659 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.814736 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.814714 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.916523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916488 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.916677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916553 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db522d78-bd16-46c0-8dbd-f6670cb42c65-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.916677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916574 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.916677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916597 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pl48\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-kube-api-access-4pl48\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.916677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916665 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.916973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916684 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.916973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916702 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.916973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916742 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.916973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916785 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.917203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.916994 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.917452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.917425 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db522d78-bd16-46c0-8dbd-f6670cb42c65-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.917583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.917544 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.917583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.917564 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.917708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.917672 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.918888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.918864 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.919226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.919207 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.938748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.938721 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:28.938867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:28.938849 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pl48\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-kube-api-access-4pl48\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:29.047049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:29.046957 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:29.385037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:29.385008 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6"] Nov 03 22:26:29.385413 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:26:29.385381 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb522d78_bd16_46c0_8dbd_f6670cb42c65.slice/crio-dc6078017b3322711da864e2aa00be560a2df8afad5fe93802bf0ff320f6e7af WatchSource:0}: Error finding container dc6078017b3322711da864e2aa00be560a2df8afad5fe93802bf0ff320f6e7af: Status 404 returned error can't find the container with id dc6078017b3322711da864e2aa00be560a2df8afad5fe93802bf0ff320f6e7af Nov 03 22:26:29.387830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:29.387801 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:29.387902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:29.387863 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:29.387902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:29.387890 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:30.206644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:30.206603 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" event={"ID":"db522d78-bd16-46c0-8dbd-f6670cb42c65","Type":"ContainerStarted","Data":"8466bb7eb798cab114c95642fe96942b0743bd92e977d1507587bb3037496e30"} Nov 03 22:26:30.206856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:30.206671 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" event={"ID":"db522d78-bd16-46c0-8dbd-f6670cb42c65","Type":"ContainerStarted","Data":"dc6078017b3322711da864e2aa00be560a2df8afad5fe93802bf0ff320f6e7af"} Nov 03 22:26:30.273057 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:30.273000 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" podStartSLOduration=2.272985518 podStartE2EDuration="2.272985518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:30.25977105 +0000 UTC m=+816.663712027" watchObservedRunningTime="2026-11-03 22:26:30.272985518 +0000 UTC m=+816.676926493" Nov 03 22:26:31.061318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:31.057970 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:31.061318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:31.058142 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:26:31.061318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:31.058191 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:26:32.063721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:32.063683 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:26:32.064333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:32.064281 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:26:33.048787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:33.048745 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:26:33.056443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:33.056398 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:26:34.065780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:34.065739 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:26:34.066339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:34.065805 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:26:35.060782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:35.060745 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 22:26:35.061081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:35.061049 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 22:26:36.360211 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:36.360178 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:36.360743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:36.360567 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:36.362012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:36.361982 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 22:26:37.350907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:37.350873 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:37.363493 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:37.363471 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:38.271936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:38.271905 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:26:40.680921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.680891 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg"] Nov 03 22:26:40.684995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.684969 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.704789 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.704761 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"openshift-service-ca.crt\"" Nov 03 22:26:40.718333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.714687 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"kube-root-ca.crt\"" Nov 03 22:26:40.718333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.714954 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"distinct-2-w0-88a4d9-kserve-self-signed-certs\"" Nov 03 22:26:40.718333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.715002 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-88a4d9\"/\"default-dockercfg-kw52f\"" Nov 03 22:26:40.718644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.718623 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg"] Nov 03 22:26:40.780884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.780849 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxv9\" (UniqueName: \"kubernetes.io/projected/c164f82e-0816-4e68-a013-aea385eb9b83-kube-api-access-rsxv9\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.781058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.781034 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-dshm\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.781101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.781067 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-home\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.781101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.781092 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-model-cache\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.781168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.781149 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-tmp-dir\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.781230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.781211 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c164f82e-0816-4e68-a013-aea385eb9b83-tls-certs\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.882955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.882916 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxv9\" (UniqueName: \"kubernetes.io/projected/c164f82e-0816-4e68-a013-aea385eb9b83-kube-api-access-rsxv9\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.883143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.883055 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-dshm\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.883143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.883091 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-home\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.883143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.883121 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-model-cache\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.883361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.883146 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-tmp-dir\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.883361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.883171 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c164f82e-0816-4e68-a013-aea385eb9b83-tls-certs\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.884070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.884041 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-home\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.884775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.884734 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-model-cache\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.885003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.884977 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-tmp-dir\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.886883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.886851 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c164f82e-0816-4e68-a013-aea385eb9b83-tls-certs\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.895601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.895571 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-dshm\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:40.906171 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:40.906136 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxv9\" (UniqueName: \"kubernetes.io/projected/c164f82e-0816-4e68-a013-aea385eb9b83-kube-api-access-rsxv9\") pod \"distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:41.001080 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:41.001004 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:41.314330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:41.314246 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg"] Nov 03 22:26:41.316401 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:26:41.316357 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc164f82e_0816_4e68_a013_aea385eb9b83.slice/crio-1cb319d21cda164dc5516df58fb5eabb2b4ea79fe08d8f69cf5d4403090cc186 WatchSource:0}: Error finding container 1cb319d21cda164dc5516df58fb5eabb2b4ea79fe08d8f69cf5d4403090cc186: Status 404 returned error can't find the container with id 1cb319d21cda164dc5516df58fb5eabb2b4ea79fe08d8f69cf5d4403090cc186 Nov 03 22:26:42.293316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:42.292369 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" event={"ID":"c164f82e-0816-4e68-a013-aea385eb9b83","Type":"ContainerStarted","Data":"779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b"} Nov 03 22:26:42.293316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:42.292427 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" event={"ID":"c164f82e-0816-4e68-a013-aea385eb9b83","Type":"ContainerStarted","Data":"1cb319d21cda164dc5516df58fb5eabb2b4ea79fe08d8f69cf5d4403090cc186"} Nov 03 22:26:42.336326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:42.336229 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podStartSLOduration=2.336208964 podStartE2EDuration="2.336208964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:42.332175852 +0000 UTC m=+828.736116841" watchObservedRunningTime="2026-11-03 22:26:42.336208964 +0000 UTC m=+828.740149940" Nov 03 22:26:43.001858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:43.001815 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:47.247219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.247175 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l"] Nov 03 22:26:47.251931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.251908 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.269500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.269473 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l"] Nov 03 22:26:47.275218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.275161 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:26:47.275543 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.275523 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:26:47.275679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.275574 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:26:47.275759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.275591 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:26:47.372919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.372889 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48wml\" (UniqueName: \"kubernetes.io/projected/322502ba-fa0c-401b-91f8-940487528ef1-kube-api-access-48wml\") pod \"maas-api-revoke-sub-bd88531e-fw92l\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.373102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.372954 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd88531e-fw92l\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.373102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.373050 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd88531e-fw92l\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.474083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.474045 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48wml\" (UniqueName: \"kubernetes.io/projected/322502ba-fa0c-401b-91f8-940487528ef1-kube-api-access-48wml\") pod \"maas-api-revoke-sub-bd88531e-fw92l\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.474393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.474107 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd88531e-fw92l\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.474393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.474144 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd88531e-fw92l\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.474847 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.474824 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd88531e-fw92l\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.476603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.476582 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd88531e-fw92l\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.495385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.495361 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48wml\" (UniqueName: \"kubernetes.io/projected/322502ba-fa0c-401b-91f8-940487528ef1-kube-api-access-48wml\") pod \"maas-api-revoke-sub-bd88531e-fw92l\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.572328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.572216 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:47.881824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:47.881793 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l"] Nov 03 22:26:47.883058 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:26:47.883030 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322502ba_fa0c_401b_91f8_940487528ef1.slice/crio-85f262b427d9d7f54f1975b871cde988f8e608fd08e2aaf9740a3a50b0f22cc2 WatchSource:0}: Error finding container 85f262b427d9d7f54f1975b871cde988f8e608fd08e2aaf9740a3a50b0f22cc2: Status 404 returned error can't find the container with id 85f262b427d9d7f54f1975b871cde988f8e608fd08e2aaf9740a3a50b0f22cc2 Nov 03 22:26:48.338025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:48.337988 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" event={"ID":"322502ba-fa0c-401b-91f8-940487528ef1","Type":"ContainerStarted","Data":"120db8bf582bc3e9fb95ae1050b84f285025e446181d879a8fcc936f627d6e59"} Nov 03 22:26:48.338450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:48.338043 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" event={"ID":"322502ba-fa0c-401b-91f8-940487528ef1","Type":"ContainerStarted","Data":"85f262b427d9d7f54f1975b871cde988f8e608fd08e2aaf9740a3a50b0f22cc2"} Nov 03 22:26:48.376873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:48.376824 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" podStartSLOduration=1.376808997 podStartE2EDuration="1.376808997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:48.373956089 +0000 UTC m=+834.777897064" watchObservedRunningTime="2026-11-03 22:26:48.376808997 +0000 UTC m=+834.780749971" Nov 03 22:26:48.385919 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:26:48.385885 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:26:49.346826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:49.346791 4903 generic.go:356] "Generic (PLEG): container finished" podID="322502ba-fa0c-401b-91f8-940487528ef1" containerID="120db8bf582bc3e9fb95ae1050b84f285025e446181d879a8fcc936f627d6e59" exitCode=0 Nov 03 22:26:49.347230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:49.346881 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" event={"ID":"322502ba-fa0c-401b-91f8-940487528ef1","Type":"ContainerDied","Data":"120db8bf582bc3e9fb95ae1050b84f285025e446181d879a8fcc936f627d6e59"} Nov 03 22:26:50.517658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.517630 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:50.613948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.613866 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322502ba-fa0c-401b-91f8-940487528ef1-kube-api-access-48wml\" (UniqueName: \"kubernetes.io/projected/322502ba-fa0c-401b-91f8-940487528ef1-kube-api-access-48wml\") pod \"322502ba-fa0c-401b-91f8-940487528ef1\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " Nov 03 22:26:50.613948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.613936 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-openshift-ca\") pod \"322502ba-fa0c-401b-91f8-940487528ef1\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " Nov 03 22:26:50.614148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.613984 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-xks-ca\") pod \"322502ba-fa0c-401b-91f8-940487528ef1\" (UID: \"322502ba-fa0c-401b-91f8-940487528ef1\") " Nov 03 22:26:50.614325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.614274 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-openshift-ca" pod "322502ba-fa0c-401b-91f8-940487528ef1" (UID: "322502ba-fa0c-401b-91f8-940487528ef1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:26:50.614584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.614568 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:26:50.616082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.616054 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-xks-ca" pod "322502ba-fa0c-401b-91f8-940487528ef1" (UID: "322502ba-fa0c-401b-91f8-940487528ef1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:26:50.616194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.616118 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322502ba-fa0c-401b-91f8-940487528ef1-kube-api-access-48wml" pod "322502ba-fa0c-401b-91f8-940487528ef1" (UID: "322502ba-fa0c-401b-91f8-940487528ef1"). InnerVolumeSpecName "kube-api-access-48wml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:26:50.715631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.715585 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48wml\" (UniqueName: \"kubernetes.io/projected/322502ba-fa0c-401b-91f8-940487528ef1-kube-api-access-48wml\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:26:50.715631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:50.715624 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322502ba-fa0c-401b-91f8-940487528ef1-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:26:51.001268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:51.001233 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:51.013671 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:51.013642 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:51.363127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:51.363032 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" Nov 03 22:26:51.363127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:51.363075 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l" event={"ID":"322502ba-fa0c-401b-91f8-940487528ef1","Type":"ContainerDied","Data":"85f262b427d9d7f54f1975b871cde988f8e608fd08e2aaf9740a3a50b0f22cc2"} Nov 03 22:26:51.363384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:51.363133 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f262b427d9d7f54f1975b871cde988f8e608fd08e2aaf9740a3a50b0f22cc2" Nov 03 22:26:51.374861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:51.374841 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:26:58.676250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.676215 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld"] Nov 03 22:26:58.676795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.676779 4903 cpu_manager.go:414] "containerMap: removing container" podUID="322502ba-fa0c-401b-91f8-940487528ef1" containerName="revoke-keys" Nov 03 22:26:58.676840 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.676796 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322502ba-fa0c-401b-91f8-940487528ef1" containerName="revoke-keys" Nov 03 22:26:58.676946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.676934 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="322502ba-fa0c-401b-91f8-940487528ef1" containerName="revoke-keys" Nov 03 22:26:58.680306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.680278 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.693640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.693621 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e19850-openshift-default-dockercfg-5fvb8\"" Nov 03 22:26:58.708701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.708674 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld"] Nov 03 22:26:58.714739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.714713 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.714871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.714749 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-496tf\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-kube-api-access-496tf\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.714871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.714778 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.714953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.714910 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.715024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.715001 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.715083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.715068 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace52e6-3c28-4087-b7c5-81685c6533db-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.715196 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.715181 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.715260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.715246 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.715324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.715309 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816154 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816120 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816170 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816196 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816236 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816252 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-496tf\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-kube-api-access-496tf\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816411 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816511 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816603 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816670 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816685 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace52e6-3c28-4087-b7c5-81685c6533db-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816733 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816769 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.816960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.816885 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.817331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.817285 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace52e6-3c28-4087-b7c5-81685c6533db-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.818733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.818709 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.818866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.818851 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.837715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.837688 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.837834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.837767 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-496tf\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-kube-api-access-496tf\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:58.994377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:58.994266 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:26:59.313032 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:26:59.312994 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ace52e6_3c28_4087_b7c5_81685c6533db.slice/crio-0c33646e10b7eee12015fbdba94d9301a18726bc20efab12415621f4fd49dc77 WatchSource:0}: Error finding container 0c33646e10b7eee12015fbdba94d9301a18726bc20efab12415621f4fd49dc77: Status 404 returned error can't find the container with id 0c33646e10b7eee12015fbdba94d9301a18726bc20efab12415621f4fd49dc77 Nov 03 22:26:59.313366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:59.313345 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld"] Nov 03 22:26:59.315546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:59.315515 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:59.315621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:59.315582 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:59.315621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:59.315608 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:26:59.424581 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:59.424545 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" event={"ID":"0ace52e6-3c28-4087-b7c5-81685c6533db","Type":"ContainerStarted","Data":"f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4"} Nov 03 22:26:59.424756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:59.424616 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" event={"ID":"0ace52e6-3c28-4087-b7c5-81685c6533db","Type":"ContainerStarted","Data":"0c33646e10b7eee12015fbdba94d9301a18726bc20efab12415621f4fd49dc77"} Nov 03 22:26:59.467497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:59.467404 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" podStartSLOduration=1.467384871 podStartE2EDuration="1.467384871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:26:59.466213105 +0000 UTC m=+845.870154080" watchObservedRunningTime="2026-11-03 22:26:59.467384871 +0000 UTC m=+845.871325849" Nov 03 22:26:59.995239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:26:59.995160 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:27:00.222117 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:00.222082 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:27:00.996813 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:00.996769 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:27:00.997276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:00.996866 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:27:01.995263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:01.995225 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:27:01.995507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:01.995310 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:27:02.995248 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:02.995211 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:27:02.995676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:02.995278 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:27:03.995135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:03.995104 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:27:03.995413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:03.995156 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:27:04.994953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:04.994912 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:27:04.995138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:04.994984 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:27:05.995134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:05.995052 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 03 22:27:05.995559 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:05.995132 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 03 22:27:06.998949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:06.998917 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:27:06.999420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:06.999209 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:27:06.999975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:06.999958 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:27:11.708746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.708710 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp"] Nov 03 22:27:11.713250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.713229 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:11.737533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.737506 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:27:11.738731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.738709 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:27:11.738938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.738921 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:27:11.739138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.739122 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:27:11.744953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.744928 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp"] Nov 03 22:27:11.844595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.844552 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57273d5a-jdkqp\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:11.844768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.844717 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnfj\" (UniqueName: \"kubernetes.io/projected/c2234ec0-095f-41f3-87ea-2040f0fac56c-kube-api-access-jcnfj\") pod \"maas-api-revoke-sub-57273d5a-jdkqp\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:11.844849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.844835 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57273d5a-jdkqp\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:11.946422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.946389 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnfj\" (UniqueName: \"kubernetes.io/projected/c2234ec0-095f-41f3-87ea-2040f0fac56c-kube-api-access-jcnfj\") pod \"maas-api-revoke-sub-57273d5a-jdkqp\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:11.946599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.946483 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57273d5a-jdkqp\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:11.946599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.946541 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57273d5a-jdkqp\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:11.947257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.947231 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57273d5a-jdkqp\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:11.948925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.948902 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57273d5a-jdkqp\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:11.968740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:11.968680 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnfj\" (UniqueName: \"kubernetes.io/projected/c2234ec0-095f-41f3-87ea-2040f0fac56c-kube-api-access-jcnfj\") pod \"maas-api-revoke-sub-57273d5a-jdkqp\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:12.028768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:12.028733 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:12.332006 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:12.331975 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2234ec0_095f_41f3_87ea_2040f0fac56c.slice/crio-dec49e9bf0a310e305f286ca6f315c5b22b8628d55b335a084a773b0b8d86f39 WatchSource:0}: Error finding container dec49e9bf0a310e305f286ca6f315c5b22b8628d55b335a084a773b0b8d86f39: Status 404 returned error can't find the container with id dec49e9bf0a310e305f286ca6f315c5b22b8628d55b335a084a773b0b8d86f39 Nov 03 22:27:12.332322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:12.332280 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp"] Nov 03 22:27:12.539318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:12.539258 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" event={"ID":"c2234ec0-095f-41f3-87ea-2040f0fac56c","Type":"ContainerStarted","Data":"dec49e9bf0a310e305f286ca6f315c5b22b8628d55b335a084a773b0b8d86f39"} Nov 03 22:27:13.556664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:13.556630 4903 generic.go:356] "Generic (PLEG): container finished" podID="c2234ec0-095f-41f3-87ea-2040f0fac56c" containerID="bc9186995207c13390ed7b38fadb1ed66e02f1425477457188ef2c0cc4cc5517" exitCode=0 Nov 03 22:27:13.557124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:13.556680 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" event={"ID":"c2234ec0-095f-41f3-87ea-2040f0fac56c","Type":"ContainerDied","Data":"bc9186995207c13390ed7b38fadb1ed66e02f1425477457188ef2c0cc4cc5517"} Nov 03 22:27:14.795822 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.795785 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:14.889988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.889898 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-openshift-ca\") pod \"c2234ec0-095f-41f3-87ea-2040f0fac56c\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " Nov 03 22:27:14.890168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.890003 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-xks-ca\") pod \"c2234ec0-095f-41f3-87ea-2040f0fac56c\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " Nov 03 22:27:14.890168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.890094 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2234ec0-095f-41f3-87ea-2040f0fac56c-kube-api-access-jcnfj\" (UniqueName: \"kubernetes.io/projected/c2234ec0-095f-41f3-87ea-2040f0fac56c-kube-api-access-jcnfj\") pod \"c2234ec0-095f-41f3-87ea-2040f0fac56c\" (UID: \"c2234ec0-095f-41f3-87ea-2040f0fac56c\") " Nov 03 22:27:14.890849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.890808 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-openshift-ca" pod "c2234ec0-095f-41f3-87ea-2040f0fac56c" (UID: "c2234ec0-095f-41f3-87ea-2040f0fac56c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:14.892677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.892653 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2234ec0-095f-41f3-87ea-2040f0fac56c-kube-api-access-jcnfj" pod "c2234ec0-095f-41f3-87ea-2040f0fac56c" (UID: "c2234ec0-095f-41f3-87ea-2040f0fac56c"). InnerVolumeSpecName "kube-api-access-jcnfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:14.893178 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.893159 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-xks-ca" pod "c2234ec0-095f-41f3-87ea-2040f0fac56c" (UID: "c2234ec0-095f-41f3-87ea-2040f0fac56c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:27:14.992028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.991989 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:14.992028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.992028 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2234ec0-095f-41f3-87ea-2040f0fac56c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:14.992242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:14.992046 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcnfj\" (UniqueName: \"kubernetes.io/projected/c2234ec0-095f-41f3-87ea-2040f0fac56c-kube-api-access-jcnfj\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:15.580785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:15.580745 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" event={"ID":"c2234ec0-095f-41f3-87ea-2040f0fac56c","Type":"ContainerDied","Data":"dec49e9bf0a310e305f286ca6f315c5b22b8628d55b335a084a773b0b8d86f39"} Nov 03 22:27:15.581063 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:15.581044 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dec49e9bf0a310e305f286ca6f315c5b22b8628d55b335a084a773b0b8d86f39" Nov 03 22:27:15.581241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:15.581225 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp" Nov 03 22:27:17.111775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.111740 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2"] Nov 03 22:27:17.112348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.112328 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c2234ec0-095f-41f3-87ea-2040f0fac56c" containerName="revoke-keys" Nov 03 22:27:17.112413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.112350 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2234ec0-095f-41f3-87ea-2040f0fac56c" containerName="revoke-keys" Nov 03 22:27:17.112940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.112916 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2234ec0-095f-41f3-87ea-2040f0fac56c" containerName="revoke-keys" Nov 03 22:27:17.118104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.118077 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.126181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.126155 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba2b51d-q2lf2\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.126344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.126213 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lcc\" (UniqueName: \"kubernetes.io/projected/dae97f1e-444f-45a7-aef6-652078cdb910-kube-api-access-q5lcc\") pod \"maas-api-revoke-sub-4ba2b51d-q2lf2\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.126405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.126360 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba2b51d-q2lf2\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.133992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.133961 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2"] Nov 03 22:27:17.140098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.140068 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:27:17.140330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.140147 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:27:17.140421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.140179 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:27:17.141942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.141923 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:27:17.227309 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.227273 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba2b51d-q2lf2\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.227489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.227411 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba2b51d-q2lf2\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.227489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.227465 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lcc\" (UniqueName: \"kubernetes.io/projected/dae97f1e-444f-45a7-aef6-652078cdb910-kube-api-access-q5lcc\") pod \"maas-api-revoke-sub-4ba2b51d-q2lf2\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.227965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.227928 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ba2b51d-q2lf2\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.229756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.229733 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ba2b51d-q2lf2\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.248816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.248780 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lcc\" (UniqueName: \"kubernetes.io/projected/dae97f1e-444f-45a7-aef6-652078cdb910-kube-api-access-q5lcc\") pod \"maas-api-revoke-sub-4ba2b51d-q2lf2\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.436014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.435983 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:17.744478 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:17.744455 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2"] Nov 03 22:27:17.750419 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:17.750391 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae97f1e_444f_45a7_aef6_652078cdb910.slice/crio-ea38f473c01c130140fac8d2d3392c93494978d48b575bdc33f4280a3fa858db WatchSource:0}: Error finding container ea38f473c01c130140fac8d2d3392c93494978d48b575bdc33f4280a3fa858db: Status 404 returned error can't find the container with id ea38f473c01c130140fac8d2d3392c93494978d48b575bdc33f4280a3fa858db Nov 03 22:27:18.385322 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:27:18.385222 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:27:18.602252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:18.602222 4903 generic.go:356] "Generic (PLEG): container finished" podID="dae97f1e-444f-45a7-aef6-652078cdb910" containerID="0c3c85e5141f491f5152b687c4796faca344076f06c5ae3953fb21c60a19512c" exitCode=0 Nov 03 22:27:18.602416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:18.602266 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" event={"ID":"dae97f1e-444f-45a7-aef6-652078cdb910","Type":"ContainerDied","Data":"0c3c85e5141f491f5152b687c4796faca344076f06c5ae3953fb21c60a19512c"} Nov 03 22:27:18.602416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:18.602350 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" event={"ID":"dae97f1e-444f-45a7-aef6-652078cdb910","Type":"ContainerStarted","Data":"ea38f473c01c130140fac8d2d3392c93494978d48b575bdc33f4280a3fa858db"} Nov 03 22:27:19.774568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.774541 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:19.857335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.857286 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-xks-ca\") pod \"dae97f1e-444f-45a7-aef6-652078cdb910\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " Nov 03 22:27:19.857555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.857536 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-openshift-ca\") pod \"dae97f1e-444f-45a7-aef6-652078cdb910\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " Nov 03 22:27:19.857621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.857604 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae97f1e-444f-45a7-aef6-652078cdb910-kube-api-access-q5lcc\" (UniqueName: \"kubernetes.io/projected/dae97f1e-444f-45a7-aef6-652078cdb910-kube-api-access-q5lcc\") pod \"dae97f1e-444f-45a7-aef6-652078cdb910\" (UID: \"dae97f1e-444f-45a7-aef6-652078cdb910\") " Nov 03 22:27:19.857852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.857824 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-openshift-ca" pod "dae97f1e-444f-45a7-aef6-652078cdb910" (UID: "dae97f1e-444f-45a7-aef6-652078cdb910"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:19.858109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.858091 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:19.859507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.859470 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-xks-ca" pod "dae97f1e-444f-45a7-aef6-652078cdb910" (UID: "dae97f1e-444f-45a7-aef6-652078cdb910"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:27:19.859649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.859626 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae97f1e-444f-45a7-aef6-652078cdb910-kube-api-access-q5lcc" pod "dae97f1e-444f-45a7-aef6-652078cdb910" (UID: "dae97f1e-444f-45a7-aef6-652078cdb910"). InnerVolumeSpecName "kube-api-access-q5lcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:19.959772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.959741 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5lcc\" (UniqueName: \"kubernetes.io/projected/dae97f1e-444f-45a7-aef6-652078cdb910-kube-api-access-q5lcc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:19.959772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:19.959767 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae97f1e-444f-45a7-aef6-652078cdb910-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:20.616677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:20.616638 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" Nov 03 22:27:20.616862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:20.616676 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2" event={"ID":"dae97f1e-444f-45a7-aef6-652078cdb910","Type":"ContainerDied","Data":"ea38f473c01c130140fac8d2d3392c93494978d48b575bdc33f4280a3fa858db"} Nov 03 22:27:20.616862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:20.616722 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea38f473c01c130140fac8d2d3392c93494978d48b575bdc33f4280a3fa858db" Nov 03 22:27:24.585208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.585168 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4"] Nov 03 22:27:24.585763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.585749 4903 cpu_manager.go:414] "containerMap: removing container" podUID="dae97f1e-444f-45a7-aef6-652078cdb910" containerName="revoke-keys" Nov 03 22:27:24.585808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.585763 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae97f1e-444f-45a7-aef6-652078cdb910" containerName="revoke-keys" Nov 03 22:27:24.585887 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.585876 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae97f1e-444f-45a7-aef6-652078cdb910" containerName="revoke-keys" Nov 03 22:27:24.589540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.589524 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.606686 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.606658 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4"] Nov 03 22:27:24.612419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.612381 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:27:24.612532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.612446 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:27:24.612532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.612446 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:27:24.612667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.612652 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:27:24.717263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.717227 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7718741e-6s6m4\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.717461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.717268 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7718741e-6s6m4\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.717461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.717446 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxn7\" (UniqueName: \"kubernetes.io/projected/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-kube-api-access-9hxn7\") pod \"maas-api-revoke-sub-7718741e-6s6m4\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.819148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.819115 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxn7\" (UniqueName: \"kubernetes.io/projected/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-kube-api-access-9hxn7\") pod \"maas-api-revoke-sub-7718741e-6s6m4\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.819323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.819257 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7718741e-6s6m4\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.819323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.819283 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7718741e-6s6m4\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.819966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.819943 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7718741e-6s6m4\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.821544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.821520 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7718741e-6s6m4\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.840469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.840414 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxn7\" (UniqueName: \"kubernetes.io/projected/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-kube-api-access-9hxn7\") pod \"maas-api-revoke-sub-7718741e-6s6m4\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:24.903185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:24.903156 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:25.222871 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:25.222838 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0a4ac4_1a34_441c_a4d5_cd5c42b64758.slice/crio-2f9cbd6dd0ab2adc88c90867f75f88cc2e5c8d0b4886404170a7eb14b6b2a69d WatchSource:0}: Error finding container 2f9cbd6dd0ab2adc88c90867f75f88cc2e5c8d0b4886404170a7eb14b6b2a69d: Status 404 returned error can't find the container with id 2f9cbd6dd0ab2adc88c90867f75f88cc2e5c8d0b4886404170a7eb14b6b2a69d Nov 03 22:27:25.223266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:25.223169 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4"] Nov 03 22:27:25.654601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:25.654562 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" event={"ID":"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758","Type":"ContainerStarted","Data":"cf2ff261aa5fd5440d65cc4732bf8507d7786d1689e37c411692588a2c2441dd"} Nov 03 22:27:25.654999 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:25.654614 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" event={"ID":"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758","Type":"ContainerStarted","Data":"2f9cbd6dd0ab2adc88c90867f75f88cc2e5c8d0b4886404170a7eb14b6b2a69d"} Nov 03 22:27:25.692683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:25.692608 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" podStartSLOduration=1.692594523 podStartE2EDuration="1.692594523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:25.691346183 +0000 UTC m=+872.095287158" watchObservedRunningTime="2026-11-03 22:27:25.692594523 +0000 UTC m=+872.096535514" Nov 03 22:27:26.662303 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:26.662268 4903 generic.go:356] "Generic (PLEG): container finished" podID="0c0a4ac4-1a34-441c-a4d5-cd5c42b64758" containerID="cf2ff261aa5fd5440d65cc4732bf8507d7786d1689e37c411692588a2c2441dd" exitCode=0 Nov 03 22:27:26.662779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:26.662325 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" event={"ID":"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758","Type":"ContainerDied","Data":"cf2ff261aa5fd5440d65cc4732bf8507d7786d1689e37c411692588a2c2441dd"} Nov 03 22:27:27.012004 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:27.011907 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:27:27.926497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:27.926147 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:28.120352 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.120284 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-openshift-ca\") pod \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " Nov 03 22:27:28.120546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.120396 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-xks-ca\") pod \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " Nov 03 22:27:28.120546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.120439 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-kube-api-access-9hxn7\" (UniqueName: \"kubernetes.io/projected/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-kube-api-access-9hxn7\") pod \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\" (UID: \"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758\") " Nov 03 22:27:28.120782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.120755 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-openshift-ca" pod "0c0a4ac4-1a34-441c-a4d5-cd5c42b64758" (UID: "0c0a4ac4-1a34-441c-a4d5-cd5c42b64758"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:28.120902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.120873 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:28.122652 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.122618 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-xks-ca" pod "0c0a4ac4-1a34-441c-a4d5-cd5c42b64758" (UID: "0c0a4ac4-1a34-441c-a4d5-cd5c42b64758"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:27:28.123217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.123194 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-kube-api-access-9hxn7" pod "0c0a4ac4-1a34-441c-a4d5-cd5c42b64758" (UID: "0c0a4ac4-1a34-441c-a4d5-cd5c42b64758"). InnerVolumeSpecName "kube-api-access-9hxn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:28.232232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.232143 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5"] Nov 03 22:27:28.233359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.233334 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0c0a4ac4-1a34-441c-a4d5-cd5c42b64758" containerName="revoke-keys" Nov 03 22:27:28.233519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.233503 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0a4ac4-1a34-441c-a4d5-cd5c42b64758" containerName="revoke-keys" Nov 03 22:27:28.233824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.233806 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0a4ac4-1a34-441c-a4d5-cd5c42b64758" containerName="revoke-keys" Nov 03 22:27:28.240310 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.232346 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:28.240310 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.238066 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hxn7\" (UniqueName: \"kubernetes.io/projected/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758-kube-api-access-9hxn7\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:28.249438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.246061 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.256782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.256754 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5"] Nov 03 22:27:28.345428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.345392 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.346232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.346172 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.346382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.346237 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.346382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.346288 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1382513-d646-48b0-b0dd-9c525da84f0a-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.346494 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.346436 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.346494 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.346467 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.346582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.346511 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.346629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.346588 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.346682 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.346647 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmhq\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-kube-api-access-4zmhq\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.464370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.464328 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.489582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.489456 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmhq\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-kube-api-access-4zmhq\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.489953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.489915 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.490181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.490163 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.490397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.490378 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.490605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.490569 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1382513-d646-48b0-b0dd-9c525da84f0a-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.490842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.490825 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.490986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.490969 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.491144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.491128 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.491784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.491758 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-credential-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.492665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.492643 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1382513-d646-48b0-b0dd-9c525da84f0a-istiod-ca-cert\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.506320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.493344 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-data\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.509475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.509447 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-socket\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.509980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.509955 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-certs\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.516024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.514574 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-envoy\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.528406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.516748 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-podinfo\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.528406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.527880 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmhq\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-kube-api-access-4zmhq\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.533597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.533569 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-token\") pod \"e2e-worker-w2-553753-openshift-default-558949869c-fgvq5\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.614317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.613533 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:28.690574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.690538 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" event={"ID":"0c0a4ac4-1a34-441c-a4d5-cd5c42b64758","Type":"ContainerDied","Data":"2f9cbd6dd0ab2adc88c90867f75f88cc2e5c8d0b4886404170a7eb14b6b2a69d"} Nov 03 22:27:28.690838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.690819 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f9cbd6dd0ab2adc88c90867f75f88cc2e5c8d0b4886404170a7eb14b6b2a69d" Nov 03 22:27:28.691010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.690996 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4" Nov 03 22:27:28.939922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:28.939866 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh"] Nov 03 22:27:29.034318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.029623 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.034565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.034538 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.034674 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.034660 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.034765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.034750 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.034855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.034840 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.034946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.034934 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfxcq\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-kube-api-access-kfxcq\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.035038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.035026 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621fc439-b1b7-4770-bd4e-529b1fcffe22-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.035304 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.035277 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.035455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.035438 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.035533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.035481 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.059331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.047806 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh"] Nov 03 22:27:29.322871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.322639 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.322871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.322681 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.322871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.322707 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.322871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.322733 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.322871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.322759 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.322871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.322784 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfxcq\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-kube-api-access-kfxcq\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.322871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.322818 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621fc439-b1b7-4770-bd4e-529b1fcffe22-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.323399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.322960 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.323399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.323004 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.334501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.326809 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.334501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.327111 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.334501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.334449 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.335699 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.335407 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621fc439-b1b7-4770-bd4e-529b1fcffe22-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.335699 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.335675 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.361927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.361883 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.387740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.382439 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.402326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.396871 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.402326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.396971 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfxcq\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-kube-api-access-kfxcq\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.456939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.456888 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:29.521692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.521271 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5"] Nov 03 22:27:29.530579 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:29.527786 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1382513_d646_48b0_b0dd_9c525da84f0a.slice/crio-6df293fa43416402c261f3545651cd298281b643cd5103eaed7c5559be4d1ea9 WatchSource:0}: Error finding container 6df293fa43416402c261f3545651cd298281b643cd5103eaed7c5559be4d1ea9: Status 404 returned error can't find the container with id 6df293fa43416402c261f3545651cd298281b643cd5103eaed7c5559be4d1ea9 Nov 03 22:27:29.531535 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.531495 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:29.531639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.531561 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:29.531639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.531588 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:29.767316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.765376 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" event={"ID":"a1382513-d646-48b0-b0dd-9c525da84f0a","Type":"ContainerStarted","Data":"6df293fa43416402c261f3545651cd298281b643cd5103eaed7c5559be4d1ea9"} Nov 03 22:27:29.940107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.938145 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:29.940107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.938330 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:29.940107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.938417 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:29.944980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:29.944950 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh"] Nov 03 22:27:30.783001 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:30.782959 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" event={"ID":"a1382513-d646-48b0-b0dd-9c525da84f0a","Type":"ContainerStarted","Data":"324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91"} Nov 03 22:27:30.797825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:30.797784 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" event={"ID":"621fc439-b1b7-4770-bd4e-529b1fcffe22","Type":"ContainerStarted","Data":"02331e20d94cda190bdf624b4bd40bf0be04909881b04b3f1947908389166c11"} Nov 03 22:27:30.798103 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:30.798082 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" event={"ID":"621fc439-b1b7-4770-bd4e-529b1fcffe22","Type":"ContainerStarted","Data":"c2d91bba862f537b5f69f8152d35481438bb310afca08ca101042b9c89632174"} Nov 03 22:27:30.838737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:30.838676 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" podStartSLOduration=2.838660495 podStartE2EDuration="2.838660495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:30.83746183 +0000 UTC m=+877.241402806" watchObservedRunningTime="2026-11-03 22:27:30.838660495 +0000 UTC m=+877.242601469" Nov 03 22:27:30.902742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:30.902684 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" podStartSLOduration=2.9026696320000003 podStartE2EDuration="2.902669632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:30.893782984 +0000 UTC m=+877.297723971" watchObservedRunningTime="2026-11-03 22:27:30.902669632 +0000 UTC m=+877.306610607" Nov 03 22:27:31.473316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.467377 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:31.473316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.468224 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:27:31.473316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.468314 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:27:31.614580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.614542 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:31.616098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.616075 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:27:31.616177 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.616124 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:27:31.633421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.633389 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2"] Nov 03 22:27:31.637886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.637866 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.661662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.661633 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:27:31.665496 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.665469 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:27:31.666337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.666317 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:27:31.678505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.678480 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:27:31.698120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.698091 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2"] Nov 03 22:27:31.708722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.708099 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmmzh\" (UniqueName: \"kubernetes.io/projected/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-kube-api-access-fmmzh\") pod \"maas-api-revoke-sub-b24c65aa-p86c2\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.708722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.708263 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b24c65aa-p86c2\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.708722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.708375 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b24c65aa-p86c2\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.809144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.809066 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b24c65aa-p86c2\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.809314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.809149 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b24c65aa-p86c2\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.809314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.809224 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmmzh\" (UniqueName: \"kubernetes.io/projected/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-kube-api-access-fmmzh\") pod \"maas-api-revoke-sub-b24c65aa-p86c2\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.810132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.810085 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b24c65aa-p86c2\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.814339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.814316 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b24c65aa-p86c2\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.830715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.830688 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmmzh\" (UniqueName: \"kubernetes.io/projected/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-kube-api-access-fmmzh\") pod \"maas-api-revoke-sub-b24c65aa-p86c2\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:31.999263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:31.999227 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:32.395898 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:32.395800 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15627fc7_38ee_4869_b2d5_6bcd7f9300a0.slice/crio-9664c4ad3a4d23bb468e28887f37ff121c40dd8b894c0de6150ee8b51d90dd63 WatchSource:0}: Error finding container 9664c4ad3a4d23bb468e28887f37ff121c40dd8b894c0de6150ee8b51d90dd63: Status 404 returned error can't find the container with id 9664c4ad3a4d23bb468e28887f37ff121c40dd8b894c0de6150ee8b51d90dd63 Nov 03 22:27:32.396086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:32.396041 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2"] Nov 03 22:27:32.464797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:32.464764 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:27:32.465067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:32.465042 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:27:32.615050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:32.615014 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:27:32.615521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:32.615072 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:27:32.817623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:32.817587 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" event={"ID":"15627fc7-38ee-4869-b2d5-6bcd7f9300a0","Type":"ContainerStarted","Data":"9664c4ad3a4d23bb468e28887f37ff121c40dd8b894c0de6150ee8b51d90dd63"} Nov 03 22:27:33.462692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:33.462655 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:27:33.462991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:33.462960 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:27:33.614515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:33.614470 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:27:33.614655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:33.614535 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:27:33.832908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:33.832814 4903 generic.go:356] "Generic (PLEG): container finished" podID="15627fc7-38ee-4869-b2d5-6bcd7f9300a0" containerID="1e1903b6b52917b5469cc015be3eb13df7a67441d7f4d80abdabe488f04bfe6b" exitCode=0 Nov 03 22:27:33.832908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:33.832899 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" event={"ID":"15627fc7-38ee-4869-b2d5-6bcd7f9300a0","Type":"ContainerDied","Data":"1e1903b6b52917b5469cc015be3eb13df7a67441d7f4d80abdabe488f04bfe6b"} Nov 03 22:27:34.468318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:34.461729 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:27:34.468318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:34.461818 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:27:34.615034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:34.615002 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:27:34.615247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:34.615227 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:27:35.030052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.030014 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj"] Nov 03 22:27:35.035172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.035148 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.106670 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.106614 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj"] Nov 03 22:27:35.108405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.108375 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2017624a-fjjvj\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.108601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.108561 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2017624a-fjjvj\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.108930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.108910 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62gtd\" (UniqueName: \"kubernetes.io/projected/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-kube-api-access-62gtd\") pod \"maas-api-revoke-sub-2017624a-fjjvj\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.209824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.209798 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2017624a-fjjvj\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.209938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.209843 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2017624a-fjjvj\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.209989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.209959 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62gtd\" (UniqueName: \"kubernetes.io/projected/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-kube-api-access-62gtd\") pod \"maas-api-revoke-sub-2017624a-fjjvj\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.211089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.211035 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2017624a-fjjvj\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.212442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.212425 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2017624a-fjjvj\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.231997 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.231974 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62gtd\" (UniqueName: \"kubernetes.io/projected/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-kube-api-access-62gtd\") pod \"maas-api-revoke-sub-2017624a-fjjvj\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.294329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.291571 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:35.310637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.310602 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-openshift-ca\") pod \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " Nov 03 22:27:35.310801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.310650 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-kube-api-access-fmmzh\" (UniqueName: \"kubernetes.io/projected/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-kube-api-access-fmmzh\") pod \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " Nov 03 22:27:35.310801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.310708 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-xks-ca\") pod \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\" (UID: \"15627fc7-38ee-4869-b2d5-6bcd7f9300a0\") " Nov 03 22:27:35.311630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.311606 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-openshift-ca" pod "15627fc7-38ee-4869-b2d5-6bcd7f9300a0" (UID: "15627fc7-38ee-4869-b2d5-6bcd7f9300a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:35.313263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.313240 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-xks-ca" pod "15627fc7-38ee-4869-b2d5-6bcd7f9300a0" (UID: "15627fc7-38ee-4869-b2d5-6bcd7f9300a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:27:35.313360 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.313307 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-kube-api-access-fmmzh" pod "15627fc7-38ee-4869-b2d5-6bcd7f9300a0" (UID: "15627fc7-38ee-4869-b2d5-6bcd7f9300a0"). InnerVolumeSpecName "kube-api-access-fmmzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:35.415093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.412508 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:35.415093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.412543 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmmzh\" (UniqueName: \"kubernetes.io/projected/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-kube-api-access-fmmzh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:35.415093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.412557 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15627fc7-38ee-4869-b2d5-6bcd7f9300a0-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:35.420349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.419395 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:35.490356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.473349 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:27:35.490356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.473430 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:27:35.614746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.614708 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:27:35.614940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.614763 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:27:35.802698 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:35.802668 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8d38b9_0d9c_4a78_bab7_18ad9c4bb873.slice/crio-50e8b289b0811e581107017a1d7e4e44b729c6011fe442105f900bc931af7e31 WatchSource:0}: Error finding container 50e8b289b0811e581107017a1d7e4e44b729c6011fe442105f900bc931af7e31: Status 404 returned error can't find the container with id 50e8b289b0811e581107017a1d7e4e44b729c6011fe442105f900bc931af7e31 Nov 03 22:27:35.805422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.805398 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj"] Nov 03 22:27:35.907607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.904664 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" event={"ID":"15627fc7-38ee-4869-b2d5-6bcd7f9300a0","Type":"ContainerDied","Data":"9664c4ad3a4d23bb468e28887f37ff121c40dd8b894c0de6150ee8b51d90dd63"} Nov 03 22:27:35.907607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.904729 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9664c4ad3a4d23bb468e28887f37ff121c40dd8b894c0de6150ee8b51d90dd63" Nov 03 22:27:35.907607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.904807 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2" Nov 03 22:27:35.907607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.906830 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" event={"ID":"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873","Type":"ContainerStarted","Data":"50e8b289b0811e581107017a1d7e4e44b729c6011fe442105f900bc931af7e31"} Nov 03 22:27:35.928559 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.928529 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk"] Nov 03 22:27:35.929419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.929400 4903 cpu_manager.go:414] "containerMap: removing container" podUID="15627fc7-38ee-4869-b2d5-6bcd7f9300a0" containerName="revoke-keys" Nov 03 22:27:35.929480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.929423 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15627fc7-38ee-4869-b2d5-6bcd7f9300a0" containerName="revoke-keys" Nov 03 22:27:35.929673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.929657 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="15627fc7-38ee-4869-b2d5-6bcd7f9300a0" containerName="revoke-keys" Nov 03 22:27:35.934650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.934629 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:35.978318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.965526 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5187e1\"/\"distinct-2-w3-5187e1-kserve-self-signed-certs\"" Nov 03 22:27:35.996767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:35.996733 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk"] Nov 03 22:27:36.025501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.025471 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-dshm\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.025666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.025555 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-tmp-dir\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.025666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.025606 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hztvp\" (UniqueName: \"kubernetes.io/projected/eac61526-e700-48e0-9293-f561cd920d53-kube-api-access-hztvp\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.025741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.025681 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac61526-e700-48e0-9293-f561cd920d53-tls-certs\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.025772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.025745 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-model-cache\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.025803 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.025772 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-home\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.128081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.127421 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac61526-e700-48e0-9293-f561cd920d53-tls-certs\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.129682 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.129210 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-model-cache\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.129682 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.129271 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-model-cache\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.129913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.129364 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-home\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.129971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.129864 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-home\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.130198 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.130110 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-dshm\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.130477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.130446 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac61526-e700-48e0-9293-f561cd920d53-tls-certs\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.130575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.130525 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-tmp-dir\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.130654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.130603 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hztvp\" (UniqueName: \"kubernetes.io/projected/eac61526-e700-48e0-9293-f561cd920d53-kube-api-access-hztvp\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.131059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.131036 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-tmp-dir\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.132344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.132320 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-dshm\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.164139 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.164109 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hztvp\" (UniqueName: \"kubernetes.io/projected/eac61526-e700-48e0-9293-f561cd920d53-kube-api-access-hztvp\") pod \"distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.295541 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.295512 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:36.461279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.461239 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 22:27:36.461485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.461324 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 22:27:36.615128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.615087 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:27:36.615321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.615179 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:27:36.676430 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.676402 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk"] Nov 03 22:27:36.677985 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:36.677953 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeac61526_e700_48e0_9293_f561cd920d53.slice/crio-b20d5a9d825f1d4f957443ac955776ff92f4d638cf77416540190450d52ad9ba WatchSource:0}: Error finding container b20d5a9d825f1d4f957443ac955776ff92f4d638cf77416540190450d52ad9ba: Status 404 returned error can't find the container with id b20d5a9d825f1d4f957443ac955776ff92f4d638cf77416540190450d52ad9ba Nov 03 22:27:36.918138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.918101 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" event={"ID":"eac61526-e700-48e0-9293-f561cd920d53","Type":"ContainerStarted","Data":"bab7f05d07fa935659f170c50c4c670dbcf78c135cfcd0665f25ef050a9caf32"} Nov 03 22:27:36.918321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.918152 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" event={"ID":"eac61526-e700-48e0-9293-f561cd920d53","Type":"ContainerStarted","Data":"b20d5a9d825f1d4f957443ac955776ff92f4d638cf77416540190450d52ad9ba"} Nov 03 22:27:36.920143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.920111 4903 generic.go:356] "Generic (PLEG): container finished" podID="4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873" containerID="0e9c9371ec2ec2668d2494a079d2c37daf372103fd77363f5579dcf30a5a89f2" exitCode=0 Nov 03 22:27:36.920245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.920193 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" event={"ID":"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873","Type":"ContainerDied","Data":"0e9c9371ec2ec2668d2494a079d2c37daf372103fd77363f5579dcf30a5a89f2"} Nov 03 22:27:36.957608 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:36.957498 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podStartSLOduration=1.9574821500000001 podStartE2EDuration="1.95748215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:36.954821308 +0000 UTC m=+883.358762301" watchObservedRunningTime="2026-11-03 22:27:36.95748215 +0000 UTC m=+883.361423130" Nov 03 22:27:37.295701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:37.295664 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:37.619194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:37.619115 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:37.619634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:37.619618 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:37.620495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:37.620475 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:27:37.678823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:37.678787 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:37.679108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:37.679089 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:37.679920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:37.679903 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:27:38.097675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.097653 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:38.167413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.167369 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-xks-ca\") pod \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " Nov 03 22:27:38.167589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.167519 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-kube-api-access-62gtd\" (UniqueName: \"kubernetes.io/projected/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-kube-api-access-62gtd\") pod \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " Nov 03 22:27:38.167675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.167610 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-openshift-ca\") pod \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\" (UID: \"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873\") " Nov 03 22:27:38.168049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.168015 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-openshift-ca" pod "4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873" (UID: "4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:38.168251 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.168231 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:38.169516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.169490 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-xks-ca" pod "4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873" (UID: "4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:27:38.170085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.170059 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-kube-api-access-62gtd" pod "4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873" (UID: "4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873"). InnerVolumeSpecName "kube-api-access-62gtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:38.269775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.269699 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:38.269775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.269725 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62gtd\" (UniqueName: \"kubernetes.io/projected/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873-kube-api-access-62gtd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:38.938689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.938660 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" Nov 03 22:27:38.939085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.938698 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj" event={"ID":"4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873","Type":"ContainerDied","Data":"50e8b289b0811e581107017a1d7e4e44b729c6011fe442105f900bc931af7e31"} Nov 03 22:27:38.939085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:38.938747 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50e8b289b0811e581107017a1d7e4e44b729c6011fe442105f900bc931af7e31" Nov 03 22:27:43.851923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.851889 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq"] Nov 03 22:27:43.854563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.852845 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873" containerName="revoke-keys" Nov 03 22:27:43.854563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.852866 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873" containerName="revoke-keys" Nov 03 22:27:43.854563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.853027 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873" containerName="revoke-keys" Nov 03 22:27:43.857227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.857204 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:43.888878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.888840 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq"] Nov 03 22:27:43.941525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.941488 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:43.941711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.941531 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:43.941711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.941657 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:43.941835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.941749 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0239ba34-b4fd-44c7-9c54-c908dba5353e-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:43.941902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.941880 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:43.941945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.941929 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkq59\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-kube-api-access-mkq59\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:43.941999 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.941950 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:43.942052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.942034 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:43.942087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:43.942067 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.043760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.043723 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.043941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.043784 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.043941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.043828 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.043946 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0239ba34-b4fd-44c7-9c54-c908dba5353e-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044050 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044102 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkq59\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-kube-api-access-mkq59\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044141 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044162 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044193 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044232 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044254 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044462 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044729 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044634 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.044867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.044845 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0239ba34-b4fd-44c7-9c54-c908dba5353e-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.046399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.046378 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.046483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.046376 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.065196 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.065171 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.065396 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.065378 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkq59\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-kube-api-access-mkq59\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.176723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.176693 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:44.509976 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:44.509939 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf WatchSource:0}: Error finding container 8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf: Status 404 returned error can't find the container with id 8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf Nov 03 22:27:44.512033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.512009 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq"] Nov 03 22:27:44.512963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.512927 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:44.513081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.512993 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:44.513081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.513024 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:27:44.994556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.994518 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" event={"ID":"0239ba34-b4fd-44c7-9c54-c908dba5353e","Type":"ContainerStarted","Data":"6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906"} Nov 03 22:27:44.995031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:44.994575 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" event={"ID":"0239ba34-b4fd-44c7-9c54-c908dba5353e","Type":"ContainerStarted","Data":"8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf"} Nov 03 22:27:45.055321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:45.052554 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podStartSLOduration=2.052532818 podStartE2EDuration="2.052532818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:45.046807623 +0000 UTC m=+891.450748600" watchObservedRunningTime="2026-11-03 22:27:45.052532818 +0000 UTC m=+891.456473794" Nov 03 22:27:45.177085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:45.177046 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:45.178650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:45.178626 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:27:45.178757 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:45.178673 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:27:46.177837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.177793 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:27:46.178274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.177856 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:27:46.226579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.226551 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw"] Nov 03 22:27:46.230815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.230799 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.275017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.271478 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:27:46.275017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.271637 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:27:46.275017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.271792 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:27:46.275017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.271846 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:27:46.276146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.276123 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw"] Nov 03 22:27:46.295933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.295907 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:46.308527 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.308501 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:46.387628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.387592 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29584f7e-nzxdw\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.387628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.387633 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29584f7e-nzxdw\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.387842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.387821 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlh66\" (UniqueName: \"kubernetes.io/projected/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-kube-api-access-rlh66\") pod \"maas-api-revoke-sub-29584f7e-nzxdw\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.489321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.489228 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlh66\" (UniqueName: \"kubernetes.io/projected/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-kube-api-access-rlh66\") pod \"maas-api-revoke-sub-29584f7e-nzxdw\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.489490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.489360 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29584f7e-nzxdw\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.489490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.489382 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29584f7e-nzxdw\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.490207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.490178 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29584f7e-nzxdw\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.491805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.491773 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29584f7e-nzxdw\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.511803 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.511761 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlh66\" (UniqueName: \"kubernetes.io/projected/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-kube-api-access-rlh66\") pod \"maas-api-revoke-sub-29584f7e-nzxdw\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.575194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.575163 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:46.877657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:46.877623 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw"] Nov 03 22:27:46.878570 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:46.878539 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a0857d_fdfa_4ac7_a969_8d08d87aeac0.slice/crio-d46bb6f3b759cd99b1261114e13de7d2d963b03a5d9e159c0fe86a5cdb23e66b WatchSource:0}: Error finding container d46bb6f3b759cd99b1261114e13de7d2d963b03a5d9e159c0fe86a5cdb23e66b: Status 404 returned error can't find the container with id d46bb6f3b759cd99b1261114e13de7d2d963b03a5d9e159c0fe86a5cdb23e66b Nov 03 22:27:47.011858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:47.011823 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" event={"ID":"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0","Type":"ContainerStarted","Data":"d46bb6f3b759cd99b1261114e13de7d2d963b03a5d9e159c0fe86a5cdb23e66b"} Nov 03 22:27:47.023334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:47.023312 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:27:47.177246 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:47.177214 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:27:47.177439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:47.177267 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:27:48.023413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:48.023379 4903 generic.go:356] "Generic (PLEG): container finished" podID="d5a0857d-fdfa-4ac7-a969-8d08d87aeac0" containerID="6f9f94898efd1e7226406d2be2124472658bf9f010d35fd59b6e1ef44c70bad9" exitCode=0 Nov 03 22:27:48.024048 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:48.024019 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" event={"ID":"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0","Type":"ContainerDied","Data":"6f9f94898efd1e7226406d2be2124472658bf9f010d35fd59b6e1ef44c70bad9"} Nov 03 22:27:48.177734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:48.177699 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:27:48.177910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:48.177749 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:27:48.391393 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:27:48.391282 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:27:49.177372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.177061 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:27:49.177372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.177118 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:27:49.271477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.271436 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:49.325780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.325705 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-kube-api-access-rlh66\" (UniqueName: \"kubernetes.io/projected/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-kube-api-access-rlh66\") pod \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " Nov 03 22:27:49.325780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.325775 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-xks-ca\") pod \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " Nov 03 22:27:49.326007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.325989 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-openshift-ca\") pod \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\" (UID: \"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0\") " Nov 03 22:27:49.326407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.326378 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-openshift-ca" pod "d5a0857d-fdfa-4ac7-a969-8d08d87aeac0" (UID: "d5a0857d-fdfa-4ac7-a969-8d08d87aeac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:49.326549 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.326521 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:49.327985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.327954 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-kube-api-access-rlh66" pod "d5a0857d-fdfa-4ac7-a969-8d08d87aeac0" (UID: "d5a0857d-fdfa-4ac7-a969-8d08d87aeac0"). InnerVolumeSpecName "kube-api-access-rlh66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:49.328158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.328135 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-xks-ca" pod "d5a0857d-fdfa-4ac7-a969-8d08d87aeac0" (UID: "d5a0857d-fdfa-4ac7-a969-8d08d87aeac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:27:49.427438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.427403 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlh66\" (UniqueName: \"kubernetes.io/projected/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-kube-api-access-rlh66\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:49.427438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:49.427428 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:50.038218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.038128 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" event={"ID":"d5a0857d-fdfa-4ac7-a969-8d08d87aeac0","Type":"ContainerDied","Data":"d46bb6f3b759cd99b1261114e13de7d2d963b03a5d9e159c0fe86a5cdb23e66b"} Nov 03 22:27:50.038218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.038153 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw" Nov 03 22:27:50.038218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.038177 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d46bb6f3b759cd99b1261114e13de7d2d963b03a5d9e159c0fe86a5cdb23e66b" Nov 03 22:27:50.177931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.177900 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:27:50.178332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.177946 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:27:50.797250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.797217 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd"] Nov 03 22:27:50.797852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.797837 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d5a0857d-fdfa-4ac7-a969-8d08d87aeac0" containerName="revoke-keys" Nov 03 22:27:50.797893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.797853 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a0857d-fdfa-4ac7-a969-8d08d87aeac0" containerName="revoke-keys" Nov 03 22:27:50.797957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.797948 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a0857d-fdfa-4ac7-a969-8d08d87aeac0" containerName="revoke-keys" Nov 03 22:27:50.801768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.801748 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:50.820166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.820136 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd"] Nov 03 22:27:50.826773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.824231 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:27:50.826773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.824243 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:27:50.826773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.824320 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:27:50.826773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.825201 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:27:50.954962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.954931 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2vvc\" (UniqueName: \"kubernetes.io/projected/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-kube-api-access-z2vvc\") pod \"maas-api-revoke-sub-19feee9a-d5cqd\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:50.955176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.954986 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19feee9a-d5cqd\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:50.955285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:50.955260 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19feee9a-d5cqd\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:51.056397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.056283 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19feee9a-d5cqd\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:51.056589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.056410 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vvc\" (UniqueName: \"kubernetes.io/projected/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-kube-api-access-z2vvc\") pod \"maas-api-revoke-sub-19feee9a-d5cqd\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:51.056589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.056435 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19feee9a-d5cqd\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:51.056940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.056918 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19feee9a-d5cqd\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:51.058746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.058721 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19feee9a-d5cqd\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:51.080798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.080758 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vvc\" (UniqueName: \"kubernetes.io/projected/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-kube-api-access-z2vvc\") pod \"maas-api-revoke-sub-19feee9a-d5cqd\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:51.122865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.122828 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:51.177858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.177821 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 22:27:51.178029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.177884 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 22:27:51.441267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.441228 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd"] Nov 03 22:27:51.441421 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:51.441389 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e54eaf8_3ac0_44c2_8a4c_41c26432067f.slice/crio-5851e945c08f4906f79d0c9b97588192df57cfdacee7692f26dd349d21203127 WatchSource:0}: Error finding container 5851e945c08f4906f79d0c9b97588192df57cfdacee7692f26dd349d21203127: Status 404 returned error can't find the container with id 5851e945c08f4906f79d0c9b97588192df57cfdacee7692f26dd349d21203127 Nov 03 22:27:51.758278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.758205 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf"] Nov 03 22:27:51.764027 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.764001 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:51.782332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.782198 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf"] Nov 03 22:27:51.867833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.867804 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxjf\" (UniqueName: \"kubernetes.io/projected/20505c61-dd5a-4fb9-81f2-25450e159897-kube-api-access-zvxjf\") pod \"maas-api-revoke-sub-a43e5677-fjmvf\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:51.868370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.868340 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a43e5677-fjmvf\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:51.868497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.868419 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a43e5677-fjmvf\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:51.969578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.969546 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a43e5677-fjmvf\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:51.969755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.969595 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a43e5677-fjmvf\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:51.969755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.969648 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxjf\" (UniqueName: \"kubernetes.io/projected/20505c61-dd5a-4fb9-81f2-25450e159897-kube-api-access-zvxjf\") pod \"maas-api-revoke-sub-a43e5677-fjmvf\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:51.970425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.970406 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a43e5677-fjmvf\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:51.971956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.971937 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a43e5677-fjmvf\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:51.991278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:51.991256 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxjf\" (UniqueName: \"kubernetes.io/projected/20505c61-dd5a-4fb9-81f2-25450e159897-kube-api-access-zvxjf\") pod \"maas-api-revoke-sub-a43e5677-fjmvf\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:52.063423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:52.063339 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" event={"ID":"8e54eaf8-3ac0-44c2-8a4c-41c26432067f","Type":"ContainerStarted","Data":"f416d9bcb90e2b3cc842d7b0ecb2ddfce18a1f3e496d15f815dbafc76aad00c2"} Nov 03 22:27:52.063423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:52.063393 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" event={"ID":"8e54eaf8-3ac0-44c2-8a4c-41c26432067f","Type":"ContainerStarted","Data":"5851e945c08f4906f79d0c9b97588192df57cfdacee7692f26dd349d21203127"} Nov 03 22:27:52.082223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:52.082197 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:52.124426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:52.124362 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" podStartSLOduration=2.12428343 podStartE2EDuration="2.12428343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:52.122630357 +0000 UTC m=+898.526571332" watchObservedRunningTime="2026-11-03 22:27:52.12428343 +0000 UTC m=+898.528224405" Nov 03 22:27:52.182278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:52.182247 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:52.182861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:52.182840 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:52.183739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:52.183719 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:27:52.387749 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:27:52.387726 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20505c61_dd5a_4fb9_81f2_25450e159897.slice/crio-371f15260df0808c6a2d016702bebbba135bdd4be529b6f1060bd2b4e6c311e2 WatchSource:0}: Error finding container 371f15260df0808c6a2d016702bebbba135bdd4be529b6f1060bd2b4e6c311e2: Status 404 returned error can't find the container with id 371f15260df0808c6a2d016702bebbba135bdd4be529b6f1060bd2b4e6c311e2 Nov 03 22:27:52.388655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:52.388555 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf"] Nov 03 22:27:53.073794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:53.073764 4903 generic.go:356] "Generic (PLEG): container finished" podID="8e54eaf8-3ac0-44c2-8a4c-41c26432067f" containerID="f416d9bcb90e2b3cc842d7b0ecb2ddfce18a1f3e496d15f815dbafc76aad00c2" exitCode=0 Nov 03 22:27:53.073976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:53.073851 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" event={"ID":"8e54eaf8-3ac0-44c2-8a4c-41c26432067f","Type":"ContainerDied","Data":"f416d9bcb90e2b3cc842d7b0ecb2ddfce18a1f3e496d15f815dbafc76aad00c2"} Nov 03 22:27:53.075350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:53.075325 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" event={"ID":"20505c61-dd5a-4fb9-81f2-25450e159897","Type":"ContainerStarted","Data":"52c2065395850c2b3477d12c80b853e6b958f37489664e7c183ace529fb2626f"} Nov 03 22:27:53.075464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:53.075373 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" event={"ID":"20505c61-dd5a-4fb9-81f2-25450e159897","Type":"ContainerStarted","Data":"371f15260df0808c6a2d016702bebbba135bdd4be529b6f1060bd2b4e6c311e2"} Nov 03 22:27:53.149287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:53.149192 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" podStartSLOduration=2.14917711 podStartE2EDuration="2.14917711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:27:53.145477702 +0000 UTC m=+899.549418689" watchObservedRunningTime="2026-11-03 22:27:53.14917711 +0000 UTC m=+899.553118084" Nov 03 22:27:53.848201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:53.848115 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:27:53.849873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:53.849844 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:27:53.856089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:53.856065 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:27:53.857902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:53.857881 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:27:54.083273 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.083242 4903 generic.go:356] "Generic (PLEG): container finished" podID="20505c61-dd5a-4fb9-81f2-25450e159897" containerID="52c2065395850c2b3477d12c80b853e6b958f37489664e7c183ace529fb2626f" exitCode=0 Nov 03 22:27:54.083481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.083320 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" event={"ID":"20505c61-dd5a-4fb9-81f2-25450e159897","Type":"ContainerDied","Data":"52c2065395850c2b3477d12c80b853e6b958f37489664e7c183ace529fb2626f"} Nov 03 22:27:54.280598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.280568 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:54.400266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.400188 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-kube-api-access-z2vvc\" (UniqueName: \"kubernetes.io/projected/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-kube-api-access-z2vvc\") pod \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " Nov 03 22:27:54.400443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.400321 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-openshift-ca\") pod \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " Nov 03 22:27:54.400443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.400403 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-xks-ca\") pod \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\" (UID: \"8e54eaf8-3ac0-44c2-8a4c-41c26432067f\") " Nov 03 22:27:54.400745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.400714 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-openshift-ca" pod "8e54eaf8-3ac0-44c2-8a4c-41c26432067f" (UID: "8e54eaf8-3ac0-44c2-8a4c-41c26432067f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:54.402276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.402250 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-kube-api-access-z2vvc" pod "8e54eaf8-3ac0-44c2-8a4c-41c26432067f" (UID: "8e54eaf8-3ac0-44c2-8a4c-41c26432067f"). InnerVolumeSpecName "kube-api-access-z2vvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:54.402384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.402286 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-xks-ca" pod "8e54eaf8-3ac0-44c2-8a4c-41c26432067f" (UID: "8e54eaf8-3ac0-44c2-8a4c-41c26432067f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:27:54.502404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.502372 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:54.502404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.502404 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:54.502623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:54.502421 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2vvc\" (UniqueName: \"kubernetes.io/projected/8e54eaf8-3ac0-44c2-8a4c-41c26432067f-kube-api-access-z2vvc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:55.094099 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.094062 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" Nov 03 22:27:55.094551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.094100 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd" event={"ID":"8e54eaf8-3ac0-44c2-8a4c-41c26432067f","Type":"ContainerDied","Data":"5851e945c08f4906f79d0c9b97588192df57cfdacee7692f26dd349d21203127"} Nov 03 22:27:55.094551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.094146 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5851e945c08f4906f79d0c9b97588192df57cfdacee7692f26dd349d21203127" Nov 03 22:27:55.263055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.263027 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:55.416319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.416270 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-openshift-ca\") pod \"20505c61-dd5a-4fb9-81f2-25450e159897\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " Nov 03 22:27:55.416497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.416325 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20505c61-dd5a-4fb9-81f2-25450e159897-kube-api-access-zvxjf\" (UniqueName: \"kubernetes.io/projected/20505c61-dd5a-4fb9-81f2-25450e159897-kube-api-access-zvxjf\") pod \"20505c61-dd5a-4fb9-81f2-25450e159897\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " Nov 03 22:27:55.416497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.416415 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-xks-ca\") pod \"20505c61-dd5a-4fb9-81f2-25450e159897\" (UID: \"20505c61-dd5a-4fb9-81f2-25450e159897\") " Nov 03 22:27:55.416640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.416616 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-openshift-ca" pod "20505c61-dd5a-4fb9-81f2-25450e159897" (UID: "20505c61-dd5a-4fb9-81f2-25450e159897"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:27:55.416989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.416964 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:55.418405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.418381 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-xks-ca" pod "20505c61-dd5a-4fb9-81f2-25450e159897" (UID: "20505c61-dd5a-4fb9-81f2-25450e159897"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:27:55.418544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.418526 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20505c61-dd5a-4fb9-81f2-25450e159897-kube-api-access-zvxjf" pod "20505c61-dd5a-4fb9-81f2-25450e159897" (UID: "20505c61-dd5a-4fb9-81f2-25450e159897"). InnerVolumeSpecName "kube-api-access-zvxjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:27:55.518455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.518413 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxjf\" (UniqueName: \"kubernetes.io/projected/20505c61-dd5a-4fb9-81f2-25450e159897-kube-api-access-zvxjf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:55.518455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:55.518452 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20505c61-dd5a-4fb9-81f2-25450e159897-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:27:56.103325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:56.103225 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" Nov 03 22:27:56.103325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:56.103252 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf" event={"ID":"20505c61-dd5a-4fb9-81f2-25450e159897","Type":"ContainerDied","Data":"371f15260df0808c6a2d016702bebbba135bdd4be529b6f1060bd2b4e6c311e2"} Nov 03 22:27:56.103325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:56.103310 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="371f15260df0808c6a2d016702bebbba135bdd4be529b6f1060bd2b4e6c311e2" Nov 03 22:27:59.903517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:59.903468 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:27:59.957321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:27:59.954136 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.027458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.027395 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.098475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.098411 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 22:28:00.262667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.259399 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Nov 03 22:28:00.262980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.262941 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.368318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.367417 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.415326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.409843 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.478315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.477917 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.500053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.499092 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.504869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.504512 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.754317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.750170 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.959204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.955383 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:00.959204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:00.956129 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:28:01.072412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.030381 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:01.072412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.031075 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:28:01.204315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.180742 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:01.238318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.205138 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:28:01.374493 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.374382 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:01.415708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.404482 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:28:01.428808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.428780 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:28:01.438904 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.438795 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:28:01.451094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.451068 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:28:01.528064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.528035 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:28:01.748513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.748483 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89"] Nov 03 22:28:01.749267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.749247 4903 cpu_manager.go:414] "containerMap: removing container" podUID="20505c61-dd5a-4fb9-81f2-25450e159897" containerName="revoke-keys" Nov 03 22:28:01.749267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.749267 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20505c61-dd5a-4fb9-81f2-25450e159897" containerName="revoke-keys" Nov 03 22:28:01.749506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.749306 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8e54eaf8-3ac0-44c2-8a4c-41c26432067f" containerName="revoke-keys" Nov 03 22:28:01.749506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.749316 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e54eaf8-3ac0-44c2-8a4c-41c26432067f" containerName="revoke-keys" Nov 03 22:28:01.749626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.749521 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="20505c61-dd5a-4fb9-81f2-25450e159897" containerName="revoke-keys" Nov 03 22:28:01.749626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.749539 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e54eaf8-3ac0-44c2-8a4c-41c26432067f" containerName="revoke-keys" Nov 03 22:28:01.755732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.755682 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:01.762341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.762311 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89"] Nov 03 22:28:01.776929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.775606 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:28:01.776929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.775714 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:01.776929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.776730 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:28:01.782526 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.782501 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:28:01.791321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.789362 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl"] Nov 03 22:28:01.812774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.812743 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl"] Nov 03 22:28:01.812935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.812886 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.825597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.825572 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzk9g\" (UniqueName: \"kubernetes.io/projected/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-kube-api-access-hzk9g\") pod \"maas-api-revoke-sub-133efb01-pjm89\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:01.825718 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.825616 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4360e7-mnhwl\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.825816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.825801 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xlr\" (UniqueName: \"kubernetes.io/projected/8cf0abb6-faaa-4246-8586-ba38c593260c-kube-api-access-j8xlr\") pod \"maas-api-revoke-sub-cb4360e7-mnhwl\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.825884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.825830 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-133efb01-pjm89\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:01.826046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.825976 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4360e7-mnhwl\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.826148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.826119 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-133efb01-pjm89\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:01.927497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.927461 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4360e7-mnhwl\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.927683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.927557 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-133efb01-pjm89\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:01.927683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.927620 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzk9g\" (UniqueName: \"kubernetes.io/projected/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-kube-api-access-hzk9g\") pod \"maas-api-revoke-sub-133efb01-pjm89\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:01.927683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.927654 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4360e7-mnhwl\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.927854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.927787 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xlr\" (UniqueName: \"kubernetes.io/projected/8cf0abb6-faaa-4246-8586-ba38c593260c-kube-api-access-j8xlr\") pod \"maas-api-revoke-sub-cb4360e7-mnhwl\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.927854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.927810 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-133efb01-pjm89\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:01.928422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.928392 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4360e7-mnhwl\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.928529 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.928493 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-133efb01-pjm89\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:01.929924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.929900 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4360e7-mnhwl\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.930134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.930117 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-133efb01-pjm89\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:01.949204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.949174 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xlr\" (UniqueName: \"kubernetes.io/projected/8cf0abb6-faaa-4246-8586-ba38c593260c-kube-api-access-j8xlr\") pod \"maas-api-revoke-sub-cb4360e7-mnhwl\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:01.949392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:01.949246 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzk9g\" (UniqueName: \"kubernetes.io/projected/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-kube-api-access-hzk9g\") pod \"maas-api-revoke-sub-133efb01-pjm89\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:02.083108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:02.083016 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:02.132507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:02.132469 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:02.402505 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:02.402469 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0ab02e_7a6c_4fe3_9c3f_fb2160e25465.slice/crio-7b19d411340f2dd33ad46a2c48a363bb5f286b85670ebfcf5ff33acdedd4b5a8 WatchSource:0}: Error finding container 7b19d411340f2dd33ad46a2c48a363bb5f286b85670ebfcf5ff33acdedd4b5a8: Status 404 returned error can't find the container with id 7b19d411340f2dd33ad46a2c48a363bb5f286b85670ebfcf5ff33acdedd4b5a8 Nov 03 22:28:02.403134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:02.403109 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89"] Nov 03 22:28:02.441891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:02.441841 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" event={"ID":"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465","Type":"ContainerStarted","Data":"7b19d411340f2dd33ad46a2c48a363bb5f286b85670ebfcf5ff33acdedd4b5a8"} Nov 03 22:28:02.459156 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:02.459126 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf0abb6_faaa_4246_8586_ba38c593260c.slice/crio-5cfd2d0f744f31491b03d6d585faa8c2facf852abc4622f87bac28585c751d9e WatchSource:0}: Error finding container 5cfd2d0f744f31491b03d6d585faa8c2facf852abc4622f87bac28585c751d9e: Status 404 returned error can't find the container with id 5cfd2d0f744f31491b03d6d585faa8c2facf852abc4622f87bac28585c751d9e Nov 03 22:28:02.460228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:02.460085 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl"] Nov 03 22:28:03.454682 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:03.454644 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" event={"ID":"8cf0abb6-faaa-4246-8586-ba38c593260c","Type":"ContainerStarted","Data":"8c5826d5a18225879fc7f89b3ab220db12014a7d17ecda25c9f5991086604b46"} Nov 03 22:28:03.455148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:03.454708 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" event={"ID":"8cf0abb6-faaa-4246-8586-ba38c593260c","Type":"ContainerStarted","Data":"5cfd2d0f744f31491b03d6d585faa8c2facf852abc4622f87bac28585c751d9e"} Nov 03 22:28:03.456953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:03.456920 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" event={"ID":"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465","Type":"ContainerStarted","Data":"12f4f7cd63bc9f332a2a235c1c36b8dc06f923da61c0eec2c6b1f949fd8f3df9"} Nov 03 22:28:03.494359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:03.494279 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" podStartSLOduration=2.494265013 podStartE2EDuration="2.494265013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:03.490383406 +0000 UTC m=+909.894324393" watchObservedRunningTime="2026-11-03 22:28:03.494265013 +0000 UTC m=+909.898205987" Nov 03 22:28:04.466490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.466452 4903 generic.go:356] "Generic (PLEG): container finished" podID="8cf0abb6-faaa-4246-8586-ba38c593260c" containerID="8c5826d5a18225879fc7f89b3ab220db12014a7d17ecda25c9f5991086604b46" exitCode=0 Nov 03 22:28:04.466913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.466529 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" event={"ID":"8cf0abb6-faaa-4246-8586-ba38c593260c","Type":"ContainerDied","Data":"8c5826d5a18225879fc7f89b3ab220db12014a7d17ecda25c9f5991086604b46"} Nov 03 22:28:04.468114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.468092 4903 generic.go:356] "Generic (PLEG): container finished" podID="5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465" containerID="12f4f7cd63bc9f332a2a235c1c36b8dc06f923da61c0eec2c6b1f949fd8f3df9" exitCode=0 Nov 03 22:28:04.468277 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.468171 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" event={"ID":"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465","Type":"ContainerDied","Data":"12f4f7cd63bc9f332a2a235c1c36b8dc06f923da61c0eec2c6b1f949fd8f3df9"} Nov 03 22:28:04.621433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.621408 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:04.676595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.676566 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-xks-ca\") pod \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " Nov 03 22:28:04.676775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.676659 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-kube-api-access-hzk9g\" (UniqueName: \"kubernetes.io/projected/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-kube-api-access-hzk9g\") pod \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " Nov 03 22:28:04.676775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.676693 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-openshift-ca\") pod \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\" (UID: \"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465\") " Nov 03 22:28:04.677162 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.677133 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-openshift-ca" pod "5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465" (UID: "5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:04.677280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.677246 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:04.679332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.679284 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-xks-ca" pod "5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465" (UID: "5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:04.679332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.679320 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-kube-api-access-hzk9g" pod "5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465" (UID: "5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465"). InnerVolumeSpecName "kube-api-access-hzk9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:04.778577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.778357 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzk9g\" (UniqueName: \"kubernetes.io/projected/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-kube-api-access-hzk9g\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:04.778577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:04.778393 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:05.476703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.476672 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" Nov 03 22:28:05.477158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.476725 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89" event={"ID":"5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465","Type":"ContainerDied","Data":"7b19d411340f2dd33ad46a2c48a363bb5f286b85670ebfcf5ff33acdedd4b5a8"} Nov 03 22:28:05.477158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.476784 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b19d411340f2dd33ad46a2c48a363bb5f286b85670ebfcf5ff33acdedd4b5a8" Nov 03 22:28:05.642081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.642048 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:05.697318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.692275 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf0abb6-faaa-4246-8586-ba38c593260c-kube-api-access-j8xlr\" (UniqueName: \"kubernetes.io/projected/8cf0abb6-faaa-4246-8586-ba38c593260c-kube-api-access-j8xlr\") pod \"8cf0abb6-faaa-4246-8586-ba38c593260c\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " Nov 03 22:28:05.697318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.692440 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-openshift-ca\") pod \"8cf0abb6-faaa-4246-8586-ba38c593260c\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " Nov 03 22:28:05.697318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.692554 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-xks-ca\") pod \"8cf0abb6-faaa-4246-8586-ba38c593260c\" (UID: \"8cf0abb6-faaa-4246-8586-ba38c593260c\") " Nov 03 22:28:05.697318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.693576 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-openshift-ca" pod "8cf0abb6-faaa-4246-8586-ba38c593260c" (UID: "8cf0abb6-faaa-4246-8586-ba38c593260c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:05.697318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.696143 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf0abb6-faaa-4246-8586-ba38c593260c-kube-api-access-j8xlr" pod "8cf0abb6-faaa-4246-8586-ba38c593260c" (UID: "8cf0abb6-faaa-4246-8586-ba38c593260c"). InnerVolumeSpecName "kube-api-access-j8xlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:05.697318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.696569 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-xks-ca" pod "8cf0abb6-faaa-4246-8586-ba38c593260c" (UID: "8cf0abb6-faaa-4246-8586-ba38c593260c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:05.794668 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.794624 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:05.794668 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.794656 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cf0abb6-faaa-4246-8586-ba38c593260c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:05.794668 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.794666 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8xlr\" (UniqueName: \"kubernetes.io/projected/8cf0abb6-faaa-4246-8586-ba38c593260c-kube-api-access-j8xlr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:05.902808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.902769 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj"] Nov 03 22:28:05.903762 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.903737 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465" containerName="revoke-keys" Nov 03 22:28:05.903762 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.903760 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465" containerName="revoke-keys" Nov 03 22:28:05.903953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.903799 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8cf0abb6-faaa-4246-8586-ba38c593260c" containerName="revoke-keys" Nov 03 22:28:05.903953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.903807 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf0abb6-faaa-4246-8586-ba38c593260c" containerName="revoke-keys" Nov 03 22:28:05.904076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.903968 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465" containerName="revoke-keys" Nov 03 22:28:05.904076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.903985 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf0abb6-faaa-4246-8586-ba38c593260c" containerName="revoke-keys" Nov 03 22:28:05.907788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.907769 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:05.924506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.924471 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj"] Nov 03 22:28:05.996578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.996489 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f6e76e4-d2jwj\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:05.996738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.996624 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcrvq\" (UniqueName: \"kubernetes.io/projected/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-kube-api-access-jcrvq\") pod \"maas-api-revoke-sub-0f6e76e4-d2jwj\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:05.996738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:05.996686 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f6e76e4-d2jwj\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:06.097881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.097837 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f6e76e4-d2jwj\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:06.098065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.098047 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f6e76e4-d2jwj\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:06.098167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.098152 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcrvq\" (UniqueName: \"kubernetes.io/projected/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-kube-api-access-jcrvq\") pod \"maas-api-revoke-sub-0f6e76e4-d2jwj\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:06.098716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.098684 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f6e76e4-d2jwj\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:06.100369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.100345 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f6e76e4-d2jwj\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:06.119731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.119700 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcrvq\" (UniqueName: \"kubernetes.io/projected/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-kube-api-access-jcrvq\") pod \"maas-api-revoke-sub-0f6e76e4-d2jwj\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:06.223094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.223053 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:06.297015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.296987 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx"] Nov 03 22:28:06.303197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.303177 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.320702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.320661 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx"] Nov 03 22:28:06.402229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.402192 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88290bcd-vgxhx\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.402428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.402372 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5q4v\" (UniqueName: \"kubernetes.io/projected/74d75dc8-035d-4aed-a647-b878a5f93536-kube-api-access-n5q4v\") pod \"maas-api-revoke-sub-88290bcd-vgxhx\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.402537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.402514 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88290bcd-vgxhx\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.485815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.485780 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" event={"ID":"8cf0abb6-faaa-4246-8586-ba38c593260c","Type":"ContainerDied","Data":"5cfd2d0f744f31491b03d6d585faa8c2facf852abc4622f87bac28585c751d9e"} Nov 03 22:28:06.486190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.485819 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl" Nov 03 22:28:06.486190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.485837 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cfd2d0f744f31491b03d6d585faa8c2facf852abc4622f87bac28585c751d9e" Nov 03 22:28:06.504594 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.504558 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88290bcd-vgxhx\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.504775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.504683 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88290bcd-vgxhx\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.504850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.504786 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5q4v\" (UniqueName: \"kubernetes.io/projected/74d75dc8-035d-4aed-a647-b878a5f93536-kube-api-access-n5q4v\") pod \"maas-api-revoke-sub-88290bcd-vgxhx\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.505345 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.505288 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88290bcd-vgxhx\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.507272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.507246 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88290bcd-vgxhx\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.527100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.527035 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5q4v\" (UniqueName: \"kubernetes.io/projected/74d75dc8-035d-4aed-a647-b878a5f93536-kube-api-access-n5q4v\") pod \"maas-api-revoke-sub-88290bcd-vgxhx\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.533285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.533260 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj"] Nov 03 22:28:06.536781 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:06.536756 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0e9f0f_37bf_4d76_ab40_8e1b1d1cf812.slice/crio-e502cb4a4257ae35e78d544390b5fe8e68802abe6f00fa8d449046b2e261e32a WatchSource:0}: Error finding container e502cb4a4257ae35e78d544390b5fe8e68802abe6f00fa8d449046b2e261e32a: Status 404 returned error can't find the container with id e502cb4a4257ae35e78d544390b5fe8e68802abe6f00fa8d449046b2e261e32a Nov 03 22:28:06.622445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.622416 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:06.928917 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:06.928889 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d75dc8_035d_4aed_a647_b878a5f93536.slice/crio-2a2f191862520c4f0016cdfe74d2ab85afd22174cdde607c2b6c7818d320c1a2 WatchSource:0}: Error finding container 2a2f191862520c4f0016cdfe74d2ab85afd22174cdde607c2b6c7818d320c1a2: Status 404 returned error can't find the container with id 2a2f191862520c4f0016cdfe74d2ab85afd22174cdde607c2b6c7818d320c1a2 Nov 03 22:28:06.929469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:06.929443 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx"] Nov 03 22:28:07.495338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:07.495242 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" event={"ID":"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812","Type":"ContainerStarted","Data":"848133600f43db0b5f68cc2cc5419ad21d2eea214a831702e9a634aeab39f644"} Nov 03 22:28:07.495759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:07.495351 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" event={"ID":"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812","Type":"ContainerStarted","Data":"e502cb4a4257ae35e78d544390b5fe8e68802abe6f00fa8d449046b2e261e32a"} Nov 03 22:28:07.497127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:07.497094 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" event={"ID":"74d75dc8-035d-4aed-a647-b878a5f93536","Type":"ContainerStarted","Data":"2cd878d2acf2ecf1b1174b855be91c1f972d7cff46df1bb6f1f8360050526980"} Nov 03 22:28:07.497259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:07.497150 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" event={"ID":"74d75dc8-035d-4aed-a647-b878a5f93536","Type":"ContainerStarted","Data":"2a2f191862520c4f0016cdfe74d2ab85afd22174cdde607c2b6c7818d320c1a2"} Nov 03 22:28:07.535754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:07.535702 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" podStartSLOduration=2.535689352 podStartE2EDuration="2.535689352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:07.533953289 +0000 UTC m=+913.937894264" watchObservedRunningTime="2026-11-03 22:28:07.535689352 +0000 UTC m=+913.939630326" Nov 03 22:28:07.573913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:07.573861 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" podStartSLOduration=1.573846879 podStartE2EDuration="1.573846879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:07.569931264 +0000 UTC m=+913.973872238" watchObservedRunningTime="2026-11-03 22:28:07.573846879 +0000 UTC m=+913.977787885" Nov 03 22:28:08.509365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:08.509333 4903 generic.go:356] "Generic (PLEG): container finished" podID="1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812" containerID="848133600f43db0b5f68cc2cc5419ad21d2eea214a831702e9a634aeab39f644" exitCode=0 Nov 03 22:28:08.509852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:08.509414 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" event={"ID":"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812","Type":"ContainerDied","Data":"848133600f43db0b5f68cc2cc5419ad21d2eea214a831702e9a634aeab39f644"} Nov 03 22:28:08.511236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:08.511209 4903 generic.go:356] "Generic (PLEG): container finished" podID="74d75dc8-035d-4aed-a647-b878a5f93536" containerID="2cd878d2acf2ecf1b1174b855be91c1f972d7cff46df1bb6f1f8360050526980" exitCode=0 Nov 03 22:28:08.511370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:08.511323 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" event={"ID":"74d75dc8-035d-4aed-a647-b878a5f93536","Type":"ContainerDied","Data":"2cd878d2acf2ecf1b1174b855be91c1f972d7cff46df1bb6f1f8360050526980"} Nov 03 22:28:09.719961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.719935 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:09.723115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.723053 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:09.847668 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.847593 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d75dc8-035d-4aed-a647-b878a5f93536-kube-api-access-n5q4v\" (UniqueName: \"kubernetes.io/projected/74d75dc8-035d-4aed-a647-b878a5f93536-kube-api-access-n5q4v\") pod \"74d75dc8-035d-4aed-a647-b878a5f93536\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " Nov 03 22:28:09.847798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.847719 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-openshift-ca\") pod \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " Nov 03 22:28:09.847798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.847739 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-xks-ca\") pod \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " Nov 03 22:28:09.847874 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.847802 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-openshift-ca\") pod \"74d75dc8-035d-4aed-a647-b878a5f93536\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " Nov 03 22:28:09.847909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.847896 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-kube-api-access-jcrvq\" (UniqueName: \"kubernetes.io/projected/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-kube-api-access-jcrvq\") pod \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\" (UID: \"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812\") " Nov 03 22:28:09.847967 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.847952 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-xks-ca\") pod \"74d75dc8-035d-4aed-a647-b878a5f93536\" (UID: \"74d75dc8-035d-4aed-a647-b878a5f93536\") " Nov 03 22:28:09.848077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.848048 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-openshift-ca" pod "1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812" (UID: "1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:09.848245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.848208 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-openshift-ca" pod "74d75dc8-035d-4aed-a647-b878a5f93536" (UID: "74d75dc8-035d-4aed-a647-b878a5f93536"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:09.848593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.848575 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:09.848706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.848599 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:09.849899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.849875 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-xks-ca" pod "1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812" (UID: "1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:09.850014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.849998 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-kube-api-access-jcrvq" pod "1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812" (UID: "1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812"). InnerVolumeSpecName "kube-api-access-jcrvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:09.850072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.850013 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-xks-ca" pod "74d75dc8-035d-4aed-a647-b878a5f93536" (UID: "74d75dc8-035d-4aed-a647-b878a5f93536"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:09.850072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.850014 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d75dc8-035d-4aed-a647-b878a5f93536-kube-api-access-n5q4v" pod "74d75dc8-035d-4aed-a647-b878a5f93536" (UID: "74d75dc8-035d-4aed-a647-b878a5f93536"). InnerVolumeSpecName "kube-api-access-n5q4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:09.949643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.949611 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcrvq\" (UniqueName: \"kubernetes.io/projected/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-kube-api-access-jcrvq\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:09.949643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.949644 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d75dc8-035d-4aed-a647-b878a5f93536-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:09.949853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.949659 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5q4v\" (UniqueName: \"kubernetes.io/projected/74d75dc8-035d-4aed-a647-b878a5f93536-kube-api-access-n5q4v\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:09.949853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:09.949683 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:10.526676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.526647 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" Nov 03 22:28:10.526676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.526654 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj" event={"ID":"1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812","Type":"ContainerDied","Data":"e502cb4a4257ae35e78d544390b5fe8e68802abe6f00fa8d449046b2e261e32a"} Nov 03 22:28:10.526929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.526707 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e502cb4a4257ae35e78d544390b5fe8e68802abe6f00fa8d449046b2e261e32a" Nov 03 22:28:10.532076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.532048 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" Nov 03 22:28:10.532227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.532162 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx" event={"ID":"74d75dc8-035d-4aed-a647-b878a5f93536","Type":"ContainerDied","Data":"2a2f191862520c4f0016cdfe74d2ab85afd22174cdde607c2b6c7818d320c1a2"} Nov 03 22:28:10.532380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.532255 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a2f191862520c4f0016cdfe74d2ab85afd22174cdde607c2b6c7818d320c1a2" Nov 03 22:28:10.827689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.827602 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g"] Nov 03 22:28:10.828418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.828398 4903 cpu_manager.go:414] "containerMap: removing container" podUID="74d75dc8-035d-4aed-a647-b878a5f93536" containerName="revoke-keys" Nov 03 22:28:10.828418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.828419 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d75dc8-035d-4aed-a647-b878a5f93536" containerName="revoke-keys" Nov 03 22:28:10.828585 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.828477 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812" containerName="revoke-keys" Nov 03 22:28:10.828585 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.828485 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812" containerName="revoke-keys" Nov 03 22:28:10.828694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.828656 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812" containerName="revoke-keys" Nov 03 22:28:10.828694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.828673 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d75dc8-035d-4aed-a647-b878a5f93536" containerName="revoke-keys" Nov 03 22:28:10.832306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.832271 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:10.850119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.850053 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:28:10.850262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.850129 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:28:10.854438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.854413 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g"] Nov 03 22:28:10.859002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.858979 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:10.859132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.859050 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:28:10.966013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.965979 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0e772e-r4g7g\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:10.966013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.966016 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72cb\" (UniqueName: \"kubernetes.io/projected/f7f83a94-43bc-4b3d-9251-2b5e53545371-kube-api-access-v72cb\") pod \"maas-api-revoke-sub-0a0e772e-r4g7g\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:10.966240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:10.966180 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0e772e-r4g7g\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:11.067204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:11.067164 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0e772e-r4g7g\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:11.067421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:11.067374 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0e772e-r4g7g\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:11.067421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:11.067410 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72cb\" (UniqueName: \"kubernetes.io/projected/f7f83a94-43bc-4b3d-9251-2b5e53545371-kube-api-access-v72cb\") pod \"maas-api-revoke-sub-0a0e772e-r4g7g\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:11.068020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:11.067997 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0e772e-r4g7g\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:11.069568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:11.069551 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0e772e-r4g7g\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:11.090440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:11.090379 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72cb\" (UniqueName: \"kubernetes.io/projected/f7f83a94-43bc-4b3d-9251-2b5e53545371-kube-api-access-v72cb\") pod \"maas-api-revoke-sub-0a0e772e-r4g7g\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:11.146131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:11.146095 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:11.455827 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:11.455797 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f83a94_43bc_4b3d_9251_2b5e53545371.slice/crio-9428a242b94fa2392abe60ff4d545f4191848824b98791a3cfa039314ef23e34 WatchSource:0}: Error finding container 9428a242b94fa2392abe60ff4d545f4191848824b98791a3cfa039314ef23e34: Status 404 returned error can't find the container with id 9428a242b94fa2392abe60ff4d545f4191848824b98791a3cfa039314ef23e34 Nov 03 22:28:11.458265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:11.458238 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g"] Nov 03 22:28:11.542049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:11.542017 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" event={"ID":"f7f83a94-43bc-4b3d-9251-2b5e53545371","Type":"ContainerStarted","Data":"9428a242b94fa2392abe60ff4d545f4191848824b98791a3cfa039314ef23e34"} Nov 03 22:28:12.550867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:12.550829 4903 generic.go:356] "Generic (PLEG): container finished" podID="f7f83a94-43bc-4b3d-9251-2b5e53545371" containerID="ceebb67c3c5dda4c494cf85ca6f6ec0923810c8ae2bbfbe982616794dabb4b43" exitCode=0 Nov 03 22:28:12.551230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:12.550912 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" event={"ID":"f7f83a94-43bc-4b3d-9251-2b5e53545371","Type":"ContainerDied","Data":"ceebb67c3c5dda4c494cf85ca6f6ec0923810c8ae2bbfbe982616794dabb4b43"} Nov 03 22:28:12.980711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:12.980676 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t"] Nov 03 22:28:12.985011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:12.984992 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.003066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.003038 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t"] Nov 03 22:28:13.096064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.096024 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72c32694-ns97t\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.096236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.096069 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbkg\" (UniqueName: \"kubernetes.io/projected/3a0cbee8-b2b0-4fac-8752-77c9853c2015-kube-api-access-kmbkg\") pod \"maas-api-revoke-sub-72c32694-ns97t\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.096236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.096208 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72c32694-ns97t\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.197973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.197935 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72c32694-ns97t\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.197973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.197972 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbkg\" (UniqueName: \"kubernetes.io/projected/3a0cbee8-b2b0-4fac-8752-77c9853c2015-kube-api-access-kmbkg\") pod \"maas-api-revoke-sub-72c32694-ns97t\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.198224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.198017 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72c32694-ns97t\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.198815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.198789 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72c32694-ns97t\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.200498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.200476 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72c32694-ns97t\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.218971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.218948 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbkg\" (UniqueName: \"kubernetes.io/projected/3a0cbee8-b2b0-4fac-8752-77c9853c2015-kube-api-access-kmbkg\") pod \"maas-api-revoke-sub-72c32694-ns97t\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.298889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.298812 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:13.614695 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:13.614666 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0cbee8_b2b0_4fac_8752_77c9853c2015.slice/crio-b7e58da5891e100289057ffbb2aa3716c481ee4fc2edce07e2ac4535203f1763 WatchSource:0}: Error finding container b7e58da5891e100289057ffbb2aa3716c481ee4fc2edce07e2ac4535203f1763: Status 404 returned error can't find the container with id b7e58da5891e100289057ffbb2aa3716c481ee4fc2edce07e2ac4535203f1763 Nov 03 22:28:13.617196 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.617169 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t"] Nov 03 22:28:13.672204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.672173 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8"] Nov 03 22:28:13.678644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.678623 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.697905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.697874 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8"] Nov 03 22:28:13.706335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.701762 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5187e1-openshift-default-dockercfg-6g9br\"" Nov 03 22:28:13.706335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.704415 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.808752 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-xks-ca\") pod \"f7f83a94-43bc-4b3d-9251-2b5e53545371\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.808957 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f83a94-43bc-4b3d-9251-2b5e53545371-kube-api-access-v72cb\" (UniqueName: \"kubernetes.io/projected/f7f83a94-43bc-4b3d-9251-2b5e53545371-kube-api-access-v72cb\") pod \"f7f83a94-43bc-4b3d-9251-2b5e53545371\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809082 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-openshift-ca\") pod \"f7f83a94-43bc-4b3d-9251-2b5e53545371\" (UID: \"f7f83a94-43bc-4b3d-9251-2b5e53545371\") " Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809313 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eabd1f-e209-4e7a-93e6-8eee2954a256-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809457 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809494 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809519 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809570 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkqg4\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-kube-api-access-lkqg4\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809642 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809714 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809754 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.809801 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.825361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.818094 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-openshift-ca" pod "f7f83a94-43bc-4b3d-9251-2b5e53545371" (UID: "f7f83a94-43bc-4b3d-9251-2b5e53545371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:13.830515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.830485 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f83a94-43bc-4b3d-9251-2b5e53545371-kube-api-access-v72cb" pod "f7f83a94-43bc-4b3d-9251-2b5e53545371" (UID: "f7f83a94-43bc-4b3d-9251-2b5e53545371"). InnerVolumeSpecName "kube-api-access-v72cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:13.831175 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.831156 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-xks-ca" pod "f7f83a94-43bc-4b3d-9251-2b5e53545371" (UID: "f7f83a94-43bc-4b3d-9251-2b5e53545371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:13.911532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.911497 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eabd1f-e209-4e7a-93e6-8eee2954a256-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.911716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.911603 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.911716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.911626 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.911716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.911642 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.911716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.911673 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkqg4\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-kube-api-access-lkqg4\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.911863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.911729 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.911903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.911784 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.912036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912012 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.912144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912120 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.912202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912185 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.912260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912131 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.912333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912267 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.912403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912362 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:13.912403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912376 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eabd1f-e209-4e7a-93e6-8eee2954a256-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.912403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912392 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f83a94-43bc-4b3d-9251-2b5e53545371-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:13.912545 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912423 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72cb\" (UniqueName: \"kubernetes.io/projected/f7f83a94-43bc-4b3d-9251-2b5e53545371-kube-api-access-v72cb\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:13.912545 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.912538 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.914321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.914285 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.914484 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.914465 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.933815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.933785 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkqg4\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-kube-api-access-lkqg4\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.933815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.933799 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:13.999859 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:13.999829 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:14.330189 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:14.330157 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35eabd1f_e209_4e7a_93e6_8eee2954a256.slice/crio-f2d34c649fc9b0d2e4fae35348591831693c505fa83fb293940032a67e3cdb8c WatchSource:0}: Error finding container f2d34c649fc9b0d2e4fae35348591831693c505fa83fb293940032a67e3cdb8c: Status 404 returned error can't find the container with id f2d34c649fc9b0d2e4fae35348591831693c505fa83fb293940032a67e3cdb8c Nov 03 22:28:14.331582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.331553 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8"] Nov 03 22:28:14.333217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.333182 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:14.333328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.333257 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:14.333328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.333317 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:14.569981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.569951 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" Nov 03 22:28:14.569981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.569968 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g" event={"ID":"f7f83a94-43bc-4b3d-9251-2b5e53545371","Type":"ContainerDied","Data":"9428a242b94fa2392abe60ff4d545f4191848824b98791a3cfa039314ef23e34"} Nov 03 22:28:14.570249 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.570023 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9428a242b94fa2392abe60ff4d545f4191848824b98791a3cfa039314ef23e34" Nov 03 22:28:14.571849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.571824 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" event={"ID":"35eabd1f-e209-4e7a-93e6-8eee2954a256","Type":"ContainerStarted","Data":"bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3"} Nov 03 22:28:14.571973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.571878 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" event={"ID":"35eabd1f-e209-4e7a-93e6-8eee2954a256","Type":"ContainerStarted","Data":"f2d34c649fc9b0d2e4fae35348591831693c505fa83fb293940032a67e3cdb8c"} Nov 03 22:28:14.573836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.573814 4903 generic.go:356] "Generic (PLEG): container finished" podID="3a0cbee8-b2b0-4fac-8752-77c9853c2015" containerID="4f8e2b1bd6525e78239e4a69c4b4abd047b6a2dd2ea1f55183e267f734f3ad19" exitCode=0 Nov 03 22:28:14.573944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.573905 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" event={"ID":"3a0cbee8-b2b0-4fac-8752-77c9853c2015","Type":"ContainerDied","Data":"4f8e2b1bd6525e78239e4a69c4b4abd047b6a2dd2ea1f55183e267f734f3ad19"} Nov 03 22:28:14.573992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.573961 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" event={"ID":"3a0cbee8-b2b0-4fac-8752-77c9853c2015","Type":"ContainerStarted","Data":"b7e58da5891e100289057ffbb2aa3716c481ee4fc2edce07e2ac4535203f1763"} Nov 03 22:28:14.617145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:14.617094 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" podStartSLOduration=1.617080358 podStartE2EDuration="1.617080358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:14.612577824 +0000 UTC m=+921.016518800" watchObservedRunningTime="2026-11-03 22:28:14.617080358 +0000 UTC m=+921.021021333" Nov 03 22:28:15.000213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.000171 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:15.002058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.002031 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:28:15.002187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.002091 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 22:28:15.810213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.810186 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:15.938987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.938958 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-xks-ca\") pod \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " Nov 03 22:28:15.939199 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.939011 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0cbee8-b2b0-4fac-8752-77c9853c2015-kube-api-access-kmbkg\" (UniqueName: \"kubernetes.io/projected/3a0cbee8-b2b0-4fac-8752-77c9853c2015-kube-api-access-kmbkg\") pod \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " Nov 03 22:28:15.939199 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.939120 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-openshift-ca\") pod \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\" (UID: \"3a0cbee8-b2b0-4fac-8752-77c9853c2015\") " Nov 03 22:28:15.939505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.939480 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-openshift-ca" pod "3a0cbee8-b2b0-4fac-8752-77c9853c2015" (UID: "3a0cbee8-b2b0-4fac-8752-77c9853c2015"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:15.967690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.966881 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-xks-ca" pod "3a0cbee8-b2b0-4fac-8752-77c9853c2015" (UID: "3a0cbee8-b2b0-4fac-8752-77c9853c2015"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:15.967690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:15.967164 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0cbee8-b2b0-4fac-8752-77c9853c2015-kube-api-access-kmbkg" pod "3a0cbee8-b2b0-4fac-8752-77c9853c2015" (UID: "3a0cbee8-b2b0-4fac-8752-77c9853c2015"). InnerVolumeSpecName "kube-api-access-kmbkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:16.000563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.000524 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:28:16.000735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.000587 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 22:28:16.047320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.040779 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:16.047320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.040814 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a0cbee8-b2b0-4fac-8752-77c9853c2015-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:16.047320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.040831 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmbkg\" (UniqueName: \"kubernetes.io/projected/3a0cbee8-b2b0-4fac-8752-77c9853c2015-kube-api-access-kmbkg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:16.066317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.057644 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92"] Nov 03 22:28:16.066317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.058584 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3a0cbee8-b2b0-4fac-8752-77c9853c2015" containerName="revoke-keys" Nov 03 22:28:16.066317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.058600 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0cbee8-b2b0-4fac-8752-77c9853c2015" containerName="revoke-keys" Nov 03 22:28:16.066317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.058637 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f7f83a94-43bc-4b3d-9251-2b5e53545371" containerName="revoke-keys" Nov 03 22:28:16.066317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.058646 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f83a94-43bc-4b3d-9251-2b5e53545371" containerName="revoke-keys" Nov 03 22:28:16.066317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.058778 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0cbee8-b2b0-4fac-8752-77c9853c2015" containerName="revoke-keys" Nov 03 22:28:16.066317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.058796 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f83a94-43bc-4b3d-9251-2b5e53545371" containerName="revoke-keys" Nov 03 22:28:16.080698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.080665 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92"] Nov 03 22:28:16.081049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.081014 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.147711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.142487 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ae46c7e-vlh92\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.147711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.142566 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2cmx\" (UniqueName: \"kubernetes.io/projected/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-kube-api-access-q2cmx\") pod \"maas-api-revoke-sub-4ae46c7e-vlh92\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.147711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.142607 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ae46c7e-vlh92\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.265427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.265178 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ae46c7e-vlh92\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.265427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.265256 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2cmx\" (UniqueName: \"kubernetes.io/projected/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-kube-api-access-q2cmx\") pod \"maas-api-revoke-sub-4ae46c7e-vlh92\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.265427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.265311 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ae46c7e-vlh92\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.266339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.266261 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ae46c7e-vlh92\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.268873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.268850 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ae46c7e-vlh92\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.288716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.288689 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2cmx\" (UniqueName: \"kubernetes.io/projected/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-kube-api-access-q2cmx\") pod \"maas-api-revoke-sub-4ae46c7e-vlh92\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.404358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.404328 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:16.591336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.591215 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" Nov 03 22:28:16.591336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.591237 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t" event={"ID":"3a0cbee8-b2b0-4fac-8752-77c9853c2015","Type":"ContainerDied","Data":"b7e58da5891e100289057ffbb2aa3716c481ee4fc2edce07e2ac4535203f1763"} Nov 03 22:28:16.591336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.591309 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e58da5891e100289057ffbb2aa3716c481ee4fc2edce07e2ac4535203f1763" Nov 03 22:28:16.737213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:16.737180 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92"] Nov 03 22:28:17.000575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:17.000539 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:28:17.001012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:17.000597 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 22:28:17.601039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:17.600999 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" event={"ID":"d3b19000-c44c-4a8e-a84e-b36b9f826e2a","Type":"ContainerStarted","Data":"6fa735968c401c70dff0e55c850deb2f149f520e32001e4bf9923edc43a0bd7f"} Nov 03 22:28:17.601218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:17.601064 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" event={"ID":"d3b19000-c44c-4a8e-a84e-b36b9f826e2a","Type":"ContainerStarted","Data":"d8fe215aae2427e1f31fab3f88e771f95485a73cdb2c6c9eb74d863cff525379"} Nov 03 22:28:17.667321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:17.664911 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" podStartSLOduration=1.6648911549999998 podStartE2EDuration="1.664891155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:17.636928698 +0000 UTC m=+924.040869673" watchObservedRunningTime="2026-11-03 22:28:17.664891155 +0000 UTC m=+924.068832130" Nov 03 22:28:18.000993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:18.000954 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:28:18.001498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:18.001197 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 22:28:18.393552 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:28:18.393513 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:28:18.611153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:18.611120 4903 generic.go:356] "Generic (PLEG): container finished" podID="d3b19000-c44c-4a8e-a84e-b36b9f826e2a" containerID="6fa735968c401c70dff0e55c850deb2f149f520e32001e4bf9923edc43a0bd7f" exitCode=0 Nov 03 22:28:18.611333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:18.611197 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" event={"ID":"d3b19000-c44c-4a8e-a84e-b36b9f826e2a","Type":"ContainerDied","Data":"6fa735968c401c70dff0e55c850deb2f149f520e32001e4bf9923edc43a0bd7f"} Nov 03 22:28:19.000631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:19.000599 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:28:19.000809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:19.000662 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 22:28:19.837373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:19.837348 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:20.000513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.000476 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:28:20.000694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.000552 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 22:28:20.024246 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.024213 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-openshift-ca\") pod \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " Nov 03 22:28:20.024448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.024430 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-kube-api-access-q2cmx\" (UniqueName: \"kubernetes.io/projected/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-kube-api-access-q2cmx\") pod \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " Nov 03 22:28:20.024558 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.024539 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-xks-ca\") pod \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\" (UID: \"d3b19000-c44c-4a8e-a84e-b36b9f826e2a\") " Nov 03 22:28:20.024645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.024622 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-openshift-ca" pod "d3b19000-c44c-4a8e-a84e-b36b9f826e2a" (UID: "d3b19000-c44c-4a8e-a84e-b36b9f826e2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:20.024988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.024966 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:20.026475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.026451 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-kube-api-access-q2cmx" pod "d3b19000-c44c-4a8e-a84e-b36b9f826e2a" (UID: "d3b19000-c44c-4a8e-a84e-b36b9f826e2a"). InnerVolumeSpecName "kube-api-access-q2cmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:20.026575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.026486 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-xks-ca" pod "d3b19000-c44c-4a8e-a84e-b36b9f826e2a" (UID: "d3b19000-c44c-4a8e-a84e-b36b9f826e2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:20.125882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.125851 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2cmx\" (UniqueName: \"kubernetes.io/projected/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-kube-api-access-q2cmx\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:20.125882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.125878 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b19000-c44c-4a8e-a84e-b36b9f826e2a-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:20.631340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.631309 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" Nov 03 22:28:20.631623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.631323 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92" event={"ID":"d3b19000-c44c-4a8e-a84e-b36b9f826e2a","Type":"ContainerDied","Data":"d8fe215aae2427e1f31fab3f88e771f95485a73cdb2c6c9eb74d863cff525379"} Nov 03 22:28:20.631723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:20.631653 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8fe215aae2427e1f31fab3f88e771f95485a73cdb2c6c9eb74d863cff525379" Nov 03 22:28:21.000776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.000739 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 22:28:21.001287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.000801 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 22:28:21.127330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.127285 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw"] Nov 03 22:28:21.127979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.127964 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d3b19000-c44c-4a8e-a84e-b36b9f826e2a" containerName="revoke-keys" Nov 03 22:28:21.128031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.127982 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b19000-c44c-4a8e-a84e-b36b9f826e2a" containerName="revoke-keys" Nov 03 22:28:21.128148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.128137 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b19000-c44c-4a8e-a84e-b36b9f826e2a" containerName="revoke-keys" Nov 03 22:28:21.132184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.132163 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.145469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.145430 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:28:21.145644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.145481 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:28:21.149527 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.149505 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw"] Nov 03 22:28:21.154145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.154098 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:21.155779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.155755 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:28:21.240624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.240591 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6gh\" (UniqueName: \"kubernetes.io/projected/f3a3ed9e-5d91-4625-8740-f98759711b30-kube-api-access-tc6gh\") pod \"maas-api-revoke-sub-b92ca3d4-j7pzw\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.240802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.240638 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b92ca3d4-j7pzw\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.240935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.240910 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b92ca3d4-j7pzw\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.342056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.341963 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b92ca3d4-j7pzw\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.342204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.342109 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6gh\" (UniqueName: \"kubernetes.io/projected/f3a3ed9e-5d91-4625-8740-f98759711b30-kube-api-access-tc6gh\") pod \"maas-api-revoke-sub-b92ca3d4-j7pzw\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.342204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.342145 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b92ca3d4-j7pzw\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.342847 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.342827 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b92ca3d4-j7pzw\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.344523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.344499 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b92ca3d4-j7pzw\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.363800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.363766 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6gh\" (UniqueName: \"kubernetes.io/projected/f3a3ed9e-5d91-4625-8740-f98759711b30-kube-api-access-tc6gh\") pod \"maas-api-revoke-sub-b92ca3d4-j7pzw\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.452470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.452433 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:21.778552 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:21.778521 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a3ed9e_5d91_4625_8740_f98759711b30.slice/crio-7df7c6492a4da5348af021fa301fe031525694e28e13abe037c5237ffd2914bc WatchSource:0}: Error finding container 7df7c6492a4da5348af021fa301fe031525694e28e13abe037c5237ffd2914bc: Status 404 returned error can't find the container with id 7df7c6492a4da5348af021fa301fe031525694e28e13abe037c5237ffd2914bc Nov 03 22:28:21.780952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:21.780924 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw"] Nov 03 22:28:22.005020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:22.004992 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:22.005523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:22.005503 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:22.006288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:22.006258 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:28:22.654605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:22.654565 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" event={"ID":"f3a3ed9e-5d91-4625-8740-f98759711b30","Type":"ContainerStarted","Data":"84c0aaf1b4f88b1d9bf19f9f0b893f24de1bed49707492cac3316592e014d151"} Nov 03 22:28:22.654758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:22.654630 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" event={"ID":"f3a3ed9e-5d91-4625-8740-f98759711b30","Type":"ContainerStarted","Data":"7df7c6492a4da5348af021fa301fe031525694e28e13abe037c5237ffd2914bc"} Nov 03 22:28:22.737548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:22.737486 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" podStartSLOduration=1.737467119 podStartE2EDuration="1.737467119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:22.729671985 +0000 UTC m=+929.133612961" watchObservedRunningTime="2026-11-03 22:28:22.737467119 +0000 UTC m=+929.141408094" Nov 03 22:28:23.663575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:23.663546 4903 generic.go:356] "Generic (PLEG): container finished" podID="f3a3ed9e-5d91-4625-8740-f98759711b30" containerID="84c0aaf1b4f88b1d9bf19f9f0b893f24de1bed49707492cac3316592e014d151" exitCode=0 Nov 03 22:28:23.663979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:23.663629 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" event={"ID":"f3a3ed9e-5d91-4625-8740-f98759711b30","Type":"ContainerDied","Data":"84c0aaf1b4f88b1d9bf19f9f0b893f24de1bed49707492cac3316592e014d151"} Nov 03 22:28:24.867282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.867257 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:24.882908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.882883 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647"] Nov 03 22:28:24.883593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.883576 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f3a3ed9e-5d91-4625-8740-f98759711b30" containerName="revoke-keys" Nov 03 22:28:24.883641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.883595 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a3ed9e-5d91-4625-8740-f98759711b30" containerName="revoke-keys" Nov 03 22:28:24.883709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.883700 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a3ed9e-5d91-4625-8740-f98759711b30" containerName="revoke-keys" Nov 03 22:28:24.885693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.885670 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-openshift-ca\") pod \"f3a3ed9e-5d91-4625-8740-f98759711b30\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " Nov 03 22:28:24.885824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.885764 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a3ed9e-5d91-4625-8740-f98759711b30-kube-api-access-tc6gh\" (UniqueName: \"kubernetes.io/projected/f3a3ed9e-5d91-4625-8740-f98759711b30-kube-api-access-tc6gh\") pod \"f3a3ed9e-5d91-4625-8740-f98759711b30\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " Nov 03 22:28:24.885824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.885789 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-xks-ca\") pod \"f3a3ed9e-5d91-4625-8740-f98759711b30\" (UID: \"f3a3ed9e-5d91-4625-8740-f98759711b30\") " Nov 03 22:28:24.886051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.886024 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-openshift-ca" pod "f3a3ed9e-5d91-4625-8740-f98759711b30" (UID: "f3a3ed9e-5d91-4625-8740-f98759711b30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:24.886455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.886428 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:24.887286 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.887266 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:24.888096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.888073 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-xks-ca" pod "f3a3ed9e-5d91-4625-8740-f98759711b30" (UID: "f3a3ed9e-5d91-4625-8740-f98759711b30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:24.888181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.888089 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a3ed9e-5d91-4625-8740-f98759711b30-kube-api-access-tc6gh" pod "f3a3ed9e-5d91-4625-8740-f98759711b30" (UID: "f3a3ed9e-5d91-4625-8740-f98759711b30"). InnerVolumeSpecName "kube-api-access-tc6gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:24.904787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.904762 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647"] Nov 03 22:28:24.987359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.987230 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6957\" (UniqueName: \"kubernetes.io/projected/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-kube-api-access-d6957\") pod \"maas-api-revoke-sub-811e2270-sm647\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:24.987515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.987444 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811e2270-sm647\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:24.987632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.987612 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811e2270-sm647\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:24.987740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.987726 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc6gh\" (UniqueName: \"kubernetes.io/projected/f3a3ed9e-5d91-4625-8740-f98759711b30-kube-api-access-tc6gh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:24.987792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:24.987745 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a3ed9e-5d91-4625-8740-f98759711b30-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:25.088839 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.088804 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811e2270-sm647\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:25.089029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.088879 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6957\" (UniqueName: \"kubernetes.io/projected/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-kube-api-access-d6957\") pod \"maas-api-revoke-sub-811e2270-sm647\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:25.089029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.088987 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811e2270-sm647\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:25.089906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.089882 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811e2270-sm647\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:25.091494 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.091470 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811e2270-sm647\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:25.110605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.110574 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6957\" (UniqueName: \"kubernetes.io/projected/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-kube-api-access-d6957\") pod \"maas-api-revoke-sub-811e2270-sm647\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:25.233111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.233077 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:25.540048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:25.540021 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a44ef58_b8fe_4312_9e9b_04dd91e05b0d.slice/crio-e2b826547aefbf4a12363539a7dc31c085eb23d7c4c3e4527cfde48b797be254 WatchSource:0}: Error finding container e2b826547aefbf4a12363539a7dc31c085eb23d7c4c3e4527cfde48b797be254: Status 404 returned error can't find the container with id e2b826547aefbf4a12363539a7dc31c085eb23d7c4c3e4527cfde48b797be254 Nov 03 22:28:25.540781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.540702 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647"] Nov 03 22:28:25.689630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.689588 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" event={"ID":"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d","Type":"ContainerStarted","Data":"e2b826547aefbf4a12363539a7dc31c085eb23d7c4c3e4527cfde48b797be254"} Nov 03 22:28:25.691404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.691377 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" event={"ID":"f3a3ed9e-5d91-4625-8740-f98759711b30","Type":"ContainerDied","Data":"7df7c6492a4da5348af021fa301fe031525694e28e13abe037c5237ffd2914bc"} Nov 03 22:28:25.691556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.691419 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7df7c6492a4da5348af021fa301fe031525694e28e13abe037c5237ffd2914bc" Nov 03 22:28:25.691556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.691416 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw" Nov 03 22:28:25.717331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.714820 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc"] Nov 03 22:28:25.720709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.720689 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:25.761316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.758779 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc"] Nov 03 22:28:25.798232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.798149 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxm9\" (UniqueName: \"kubernetes.io/projected/31b82496-2c18-435b-a7af-9aa1e8cdec24-kube-api-access-vvxm9\") pod \"maas-api-revoke-sub-73db1173-cjdzc\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:25.798399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.798247 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73db1173-cjdzc\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:25.798483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.798419 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73db1173-cjdzc\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:25.899827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.899795 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73db1173-cjdzc\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:25.900304 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.899919 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxm9\" (UniqueName: \"kubernetes.io/projected/31b82496-2c18-435b-a7af-9aa1e8cdec24-kube-api-access-vvxm9\") pod \"maas-api-revoke-sub-73db1173-cjdzc\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:25.900304 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.900007 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73db1173-cjdzc\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:25.900636 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.900608 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73db1173-cjdzc\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:25.903868 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.903837 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73db1173-cjdzc\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:25.921251 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:25.921228 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxm9\" (UniqueName: \"kubernetes.io/projected/31b82496-2c18-435b-a7af-9aa1e8cdec24-kube-api-access-vvxm9\") pod \"maas-api-revoke-sub-73db1173-cjdzc\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:26.051956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:26.051871 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:26.366924 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:26.366894 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b82496_2c18_435b_a7af_9aa1e8cdec24.slice/crio-95ac36831635efd420a4ee9c5e104749c3c2132934127f1ac856abae4420ce1c WatchSource:0}: Error finding container 95ac36831635efd420a4ee9c5e104749c3c2132934127f1ac856abae4420ce1c: Status 404 returned error can't find the container with id 95ac36831635efd420a4ee9c5e104749c3c2132934127f1ac856abae4420ce1c Nov 03 22:28:26.368403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:26.368377 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc"] Nov 03 22:28:26.701694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:26.701662 4903 generic.go:356] "Generic (PLEG): container finished" podID="6a44ef58-b8fe-4312-9e9b-04dd91e05b0d" containerID="7553b2b341ffbca8b96e5900d0c5109a73cf140541726e436f16ea6fb8936409" exitCode=0 Nov 03 22:28:26.701902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:26.701756 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" event={"ID":"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d","Type":"ContainerDied","Data":"7553b2b341ffbca8b96e5900d0c5109a73cf140541726e436f16ea6fb8936409"} Nov 03 22:28:26.703488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:26.703461 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" event={"ID":"31b82496-2c18-435b-a7af-9aa1e8cdec24","Type":"ContainerStarted","Data":"8671cfd38dd76227dee8a76edb71ba377e454196fa3d80594a58e294a99111dc"} Nov 03 22:28:26.703594 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:26.703508 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" event={"ID":"31b82496-2c18-435b-a7af-9aa1e8cdec24","Type":"ContainerStarted","Data":"95ac36831635efd420a4ee9c5e104749c3c2132934127f1ac856abae4420ce1c"} Nov 03 22:28:26.777374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:26.777238 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" podStartSLOduration=1.777224315 podStartE2EDuration="1.777224315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:26.773585054 +0000 UTC m=+933.177526029" watchObservedRunningTime="2026-11-03 22:28:26.777224315 +0000 UTC m=+933.181165290" Nov 03 22:28:27.614459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.614362 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:28:27.713364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.713320 4903 generic.go:356] "Generic (PLEG): container finished" podID="31b82496-2c18-435b-a7af-9aa1e8cdec24" containerID="8671cfd38dd76227dee8a76edb71ba377e454196fa3d80594a58e294a99111dc" exitCode=0 Nov 03 22:28:27.713565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.713378 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" event={"ID":"31b82496-2c18-435b-a7af-9aa1e8cdec24","Type":"ContainerDied","Data":"8671cfd38dd76227dee8a76edb71ba377e454196fa3d80594a58e294a99111dc"} Nov 03 22:28:27.885900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.885835 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:27.928168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.928135 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-xks-ca\") pod \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " Nov 03 22:28:27.928375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.928230 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-kube-api-access-d6957\" (UniqueName: \"kubernetes.io/projected/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-kube-api-access-d6957\") pod \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " Nov 03 22:28:27.928375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.928281 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-openshift-ca\") pod \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\" (UID: \"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d\") " Nov 03 22:28:27.929166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.929141 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-openshift-ca" pod "6a44ef58-b8fe-4312-9e9b-04dd91e05b0d" (UID: "6a44ef58-b8fe-4312-9e9b-04dd91e05b0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:27.931193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.931167 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-kube-api-access-d6957" pod "6a44ef58-b8fe-4312-9e9b-04dd91e05b0d" (UID: "6a44ef58-b8fe-4312-9e9b-04dd91e05b0d"). InnerVolumeSpecName "kube-api-access-d6957". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:27.931411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:27.931387 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-xks-ca" pod "6a44ef58-b8fe-4312-9e9b-04dd91e05b0d" (UID: "6a44ef58-b8fe-4312-9e9b-04dd91e05b0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:28.030383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:28.030347 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:28.030383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:28.030375 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6957\" (UniqueName: \"kubernetes.io/projected/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-kube-api-access-d6957\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:28.030383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:28.030389 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:28.751851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:28.751818 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" Nov 03 22:28:28.757954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:28.757923 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647" event={"ID":"6a44ef58-b8fe-4312-9e9b-04dd91e05b0d","Type":"ContainerDied","Data":"e2b826547aefbf4a12363539a7dc31c085eb23d7c4c3e4527cfde48b797be254"} Nov 03 22:28:28.758140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:28.758124 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2b826547aefbf4a12363539a7dc31c085eb23d7c4c3e4527cfde48b797be254" Nov 03 22:28:29.356706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.347278 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:29.527375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.513789 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-xks-ca\") pod \"31b82496-2c18-435b-a7af-9aa1e8cdec24\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " Nov 03 22:28:29.527680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.527654 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-openshift-ca\") pod \"31b82496-2c18-435b-a7af-9aa1e8cdec24\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " Nov 03 22:28:29.528020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.528001 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31b82496-2c18-435b-a7af-9aa1e8cdec24-kube-api-access-vvxm9\" (UniqueName: \"kubernetes.io/projected/31b82496-2c18-435b-a7af-9aa1e8cdec24-kube-api-access-vvxm9\") pod \"31b82496-2c18-435b-a7af-9aa1e8cdec24\" (UID: \"31b82496-2c18-435b-a7af-9aa1e8cdec24\") " Nov 03 22:28:29.585548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.577762 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-openshift-ca" pod "31b82496-2c18-435b-a7af-9aa1e8cdec24" (UID: "31b82496-2c18-435b-a7af-9aa1e8cdec24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:29.598511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.598471 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-xks-ca" pod "31b82496-2c18-435b-a7af-9aa1e8cdec24" (UID: "31b82496-2c18-435b-a7af-9aa1e8cdec24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:29.657767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.657728 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:29.658010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.657993 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31b82496-2c18-435b-a7af-9aa1e8cdec24-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:29.721667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.721625 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31b82496-2c18-435b-a7af-9aa1e8cdec24-kube-api-access-vvxm9" pod "31b82496-2c18-435b-a7af-9aa1e8cdec24" (UID: "31b82496-2c18-435b-a7af-9aa1e8cdec24"). InnerVolumeSpecName "kube-api-access-vvxm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:29.866986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:29.824587 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvxm9\" (UniqueName: \"kubernetes.io/projected/31b82496-2c18-435b-a7af-9aa1e8cdec24-kube-api-access-vvxm9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:30.204377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.191098 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" Nov 03 22:28:30.235369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.229883 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc" event={"ID":"31b82496-2c18-435b-a7af-9aa1e8cdec24","Type":"ContainerDied","Data":"95ac36831635efd420a4ee9c5e104749c3c2132934127f1ac856abae4420ce1c"} Nov 03 22:28:30.235369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.229963 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95ac36831635efd420a4ee9c5e104749c3c2132934127f1ac856abae4420ce1c" Nov 03 22:28:30.264477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.260415 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.287323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.284362 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.410465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.410404 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.441422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.441317 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 03 22:28:30.466691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.466644 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.468605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.466919 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.490445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.490392 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.511490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.511436 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.521324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.512042 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 22:28:30.581442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.581396 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.746469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.746371 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.779732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.747994 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.811897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.811853 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 22:28:30.835960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.835911 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.880489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.880437 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:30.904360 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:30.903548 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:28:31.316232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.315967 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.316596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.316574 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:28:31.365319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.317161 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.434444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.430513 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.435247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.435222 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:28:31.490320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.462749 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 03 22:28:31.490320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.463161 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:28:31.490320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.476318 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 22:28:31.490320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.476852 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:28:31.490691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.490379 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.490784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.490757 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:28:31.491329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.491277 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.491649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.491626 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:28:31.513593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.513557 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.514145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.514123 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:28:31.524419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.524368 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.524980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.524963 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:28:31.587154 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.587064 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.587758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.587733 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:28:31.825320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.824630 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.827260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.827218 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:31.868388 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.850440 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:28:31.980316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:31.956794 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:28:32.307127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.307063 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 03 22:28:32.535318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.494310 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.536193 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.536227 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.567232 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.573797 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.578136 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.578179 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.587617 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.613985 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.639652 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.641678 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.641711 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.641755 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.744587 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.883984 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.884119 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.884160 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:28:32.884314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.884191 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:28:32.909150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.815932 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.909150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.873850 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.948574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.948496 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.979323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.965502 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:28:32.979323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.966189 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:28:32.979323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.974889 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:28:32.979323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.975436 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:28:32.979323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.976172 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:28:32.985470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:32.985447 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:28:33.005666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.005635 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:28:33.028963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.028938 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:28:33.052880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.052851 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:28:33.058328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.058289 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:28:33.059097 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.059082 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:28:33.076312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.074150 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:28:33.076312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.075580 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:28:33.077963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.077644 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:28:33.090110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.087201 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:28:33.090685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:33.090664 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:28:36.046218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.046125 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d"] Nov 03 22:28:36.047009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.046986 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6a44ef58-b8fe-4312-9e9b-04dd91e05b0d" containerName="revoke-keys" Nov 03 22:28:36.047105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.047010 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a44ef58-b8fe-4312-9e9b-04dd91e05b0d" containerName="revoke-keys" Nov 03 22:28:36.047105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.047041 4903 cpu_manager.go:414] "containerMap: removing container" podUID="31b82496-2c18-435b-a7af-9aa1e8cdec24" containerName="revoke-keys" Nov 03 22:28:36.047105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.047048 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31b82496-2c18-435b-a7af-9aa1e8cdec24" containerName="revoke-keys" Nov 03 22:28:36.047259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.047210 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a44ef58-b8fe-4312-9e9b-04dd91e05b0d" containerName="revoke-keys" Nov 03 22:28:36.047259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.047229 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="31b82496-2c18-435b-a7af-9aa1e8cdec24" containerName="revoke-keys" Nov 03 22:28:36.052343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.052322 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.066637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.066613 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:36.069240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.069215 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d"] Nov 03 22:28:36.072966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.072943 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:28:36.073227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.073205 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:28:36.074128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.074108 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:28:36.126375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.126339 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed0a4a3-d7c2d\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.126556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.126388 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed0a4a3-d7c2d\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.126556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.126488 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6sn\" (UniqueName: \"kubernetes.io/projected/f65bce2e-b44d-46cc-8e78-b7de172f8611-kube-api-access-hg6sn\") pod \"maas-api-revoke-sub-2ed0a4a3-d7c2d\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.227746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.227712 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed0a4a3-d7c2d\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.227746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.227750 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed0a4a3-d7c2d\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.227962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.227816 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6sn\" (UniqueName: \"kubernetes.io/projected/f65bce2e-b44d-46cc-8e78-b7de172f8611-kube-api-access-hg6sn\") pod \"maas-api-revoke-sub-2ed0a4a3-d7c2d\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.228494 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.228470 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed0a4a3-d7c2d\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.230117 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.230092 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed0a4a3-d7c2d\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.253881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.253847 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6sn\" (UniqueName: \"kubernetes.io/projected/f65bce2e-b44d-46cc-8e78-b7de172f8611-kube-api-access-hg6sn\") pod \"maas-api-revoke-sub-2ed0a4a3-d7c2d\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.377449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.369529 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:36.693468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.693435 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d"] Nov 03 22:28:36.699444 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:36.699408 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65bce2e_b44d_46cc_8e78_b7de172f8611.slice/crio-92f02a6b5bf2fce71eb2741e01f8b1a562626df4bf5c8e503b72405a00912e12 WatchSource:0}: Error finding container 92f02a6b5bf2fce71eb2741e01f8b1a562626df4bf5c8e503b72405a00912e12: Status 404 returned error can't find the container with id 92f02a6b5bf2fce71eb2741e01f8b1a562626df4bf5c8e503b72405a00912e12 Nov 03 22:28:36.862520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.862475 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d"] Nov 03 22:28:36.868057 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.868016 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:36.885278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.885249 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d"] Nov 03 22:28:36.938096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.938006 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxjtd\" (UniqueName: \"kubernetes.io/projected/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-kube-api-access-rxjtd\") pod \"maas-api-revoke-sub-510e733b-lph9d\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:36.938241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.938165 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-510e733b-lph9d\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:36.939345 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.938559 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-510e733b-lph9d\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:36.988232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.988196 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" event={"ID":"f65bce2e-b44d-46cc-8e78-b7de172f8611","Type":"ContainerStarted","Data":"021cae792cf493dc62e6a222f5fe79a28d494c1f7ea2c2615f275456e2bfa9b4"} Nov 03 22:28:36.988430 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:36.988250 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" event={"ID":"f65bce2e-b44d-46cc-8e78-b7de172f8611","Type":"ContainerStarted","Data":"92f02a6b5bf2fce71eb2741e01f8b1a562626df4bf5c8e503b72405a00912e12"} Nov 03 22:28:37.024734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.024680 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" podStartSLOduration=1.024666486 podStartE2EDuration="1.024666486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:37.023781386 +0000 UTC m=+943.427722360" watchObservedRunningTime="2026-11-03 22:28:37.024666486 +0000 UTC m=+943.428607458" Nov 03 22:28:37.040361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.040281 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-510e733b-lph9d\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:37.040551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.040390 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxjtd\" (UniqueName: \"kubernetes.io/projected/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-kube-api-access-rxjtd\") pod \"maas-api-revoke-sub-510e733b-lph9d\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:37.040551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.040508 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-510e733b-lph9d\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:37.040938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.040914 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-510e733b-lph9d\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:37.042737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.042715 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-510e733b-lph9d\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:37.061603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.061576 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxjtd\" (UniqueName: \"kubernetes.io/projected/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-kube-api-access-rxjtd\") pod \"maas-api-revoke-sub-510e733b-lph9d\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:37.187774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.187736 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:37.498361 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:37.498332 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27a4e00_abaf_47bb_a3d3_6f2f59035d1c.slice/crio-e240e2b316e19b1329a49220ddb48b8ee8f5c06a7a385a31b8bfc13d403af6f4 WatchSource:0}: Error finding container e240e2b316e19b1329a49220ddb48b8ee8f5c06a7a385a31b8bfc13d403af6f4: Status 404 returned error can't find the container with id e240e2b316e19b1329a49220ddb48b8ee8f5c06a7a385a31b8bfc13d403af6f4 Nov 03 22:28:37.500318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.500276 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d"] Nov 03 22:28:37.997910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.997869 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" event={"ID":"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c","Type":"ContainerStarted","Data":"b18388d1f6f9e526595d860bc028fcbb22f283fa457e2e56903c9fc61c739063"} Nov 03 22:28:37.998072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.997962 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" event={"ID":"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c","Type":"ContainerStarted","Data":"e240e2b316e19b1329a49220ddb48b8ee8f5c06a7a385a31b8bfc13d403af6f4"} Nov 03 22:28:37.999683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.999654 4903 generic.go:356] "Generic (PLEG): container finished" podID="f65bce2e-b44d-46cc-8e78-b7de172f8611" containerID="021cae792cf493dc62e6a222f5fe79a28d494c1f7ea2c2615f275456e2bfa9b4" exitCode=0 Nov 03 22:28:37.999790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:37.999725 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" event={"ID":"f65bce2e-b44d-46cc-8e78-b7de172f8611","Type":"ContainerDied","Data":"021cae792cf493dc62e6a222f5fe79a28d494c1f7ea2c2615f275456e2bfa9b4"} Nov 03 22:28:38.040515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:38.040413 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" podStartSLOduration=2.040395307 podStartE2EDuration="2.040395307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:38.034617671 +0000 UTC m=+944.438558646" watchObservedRunningTime="2026-11-03 22:28:38.040395307 +0000 UTC m=+944.444336283" Nov 03 22:28:39.010667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.010632 4903 generic.go:356] "Generic (PLEG): container finished" podID="f27a4e00-abaf-47bb-a3d3-6f2f59035d1c" containerID="b18388d1f6f9e526595d860bc028fcbb22f283fa457e2e56903c9fc61c739063" exitCode=0 Nov 03 22:28:39.011168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.010717 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" event={"ID":"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c","Type":"ContainerDied","Data":"b18388d1f6f9e526595d860bc028fcbb22f283fa457e2e56903c9fc61c739063"} Nov 03 22:28:39.175758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.175734 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:39.269498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.269422 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65bce2e-b44d-46cc-8e78-b7de172f8611-kube-api-access-hg6sn\" (UniqueName: \"kubernetes.io/projected/f65bce2e-b44d-46cc-8e78-b7de172f8611-kube-api-access-hg6sn\") pod \"f65bce2e-b44d-46cc-8e78-b7de172f8611\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " Nov 03 22:28:39.269632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.269572 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-openshift-ca\") pod \"f65bce2e-b44d-46cc-8e78-b7de172f8611\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " Nov 03 22:28:39.269691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.269678 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-xks-ca\") pod \"f65bce2e-b44d-46cc-8e78-b7de172f8611\" (UID: \"f65bce2e-b44d-46cc-8e78-b7de172f8611\") " Nov 03 22:28:39.269948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.269926 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-openshift-ca" pod "f65bce2e-b44d-46cc-8e78-b7de172f8611" (UID: "f65bce2e-b44d-46cc-8e78-b7de172f8611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:39.270221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.270200 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:39.271669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.271634 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65bce2e-b44d-46cc-8e78-b7de172f8611-kube-api-access-hg6sn" pod "f65bce2e-b44d-46cc-8e78-b7de172f8611" (UID: "f65bce2e-b44d-46cc-8e78-b7de172f8611"). InnerVolumeSpecName "kube-api-access-hg6sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:39.271771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.271701 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-xks-ca" pod "f65bce2e-b44d-46cc-8e78-b7de172f8611" (UID: "f65bce2e-b44d-46cc-8e78-b7de172f8611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:39.371735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.371702 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f65bce2e-b44d-46cc-8e78-b7de172f8611-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:39.371735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:39.371732 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6sn\" (UniqueName: \"kubernetes.io/projected/f65bce2e-b44d-46cc-8e78-b7de172f8611-kube-api-access-hg6sn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:40.021137 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.021056 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" Nov 03 22:28:40.021137 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.021089 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d" event={"ID":"f65bce2e-b44d-46cc-8e78-b7de172f8611","Type":"ContainerDied","Data":"92f02a6b5bf2fce71eb2741e01f8b1a562626df4bf5c8e503b72405a00912e12"} Nov 03 22:28:40.021740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.021138 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f02a6b5bf2fce71eb2741e01f8b1a562626df4bf5c8e503b72405a00912e12" Nov 03 22:28:40.191364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.191331 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:40.284856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.284769 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-kube-api-access-rxjtd\" (UniqueName: \"kubernetes.io/projected/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-kube-api-access-rxjtd\") pod \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " Nov 03 22:28:40.285038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.284921 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-xks-ca\") pod \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " Nov 03 22:28:40.285038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.285009 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-openshift-ca\") pod \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\" (UID: \"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c\") " Nov 03 22:28:40.285442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.285414 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-openshift-ca" pod "f27a4e00-abaf-47bb-a3d3-6f2f59035d1c" (UID: "f27a4e00-abaf-47bb-a3d3-6f2f59035d1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:40.285559 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.285531 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:40.286885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.286856 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-kube-api-access-rxjtd" pod "f27a4e00-abaf-47bb-a3d3-6f2f59035d1c" (UID: "f27a4e00-abaf-47bb-a3d3-6f2f59035d1c"). InnerVolumeSpecName "kube-api-access-rxjtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:40.286992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.286938 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-xks-ca" pod "f27a4e00-abaf-47bb-a3d3-6f2f59035d1c" (UID: "f27a4e00-abaf-47bb-a3d3-6f2f59035d1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:40.387135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.387099 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxjtd\" (UniqueName: \"kubernetes.io/projected/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-kube-api-access-rxjtd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:40.387135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.387131 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:40.967755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.967715 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2"] Nov 03 22:28:40.968709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.968686 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f27a4e00-abaf-47bb-a3d3-6f2f59035d1c" containerName="revoke-keys" Nov 03 22:28:40.968809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.968710 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27a4e00-abaf-47bb-a3d3-6f2f59035d1c" containerName="revoke-keys" Nov 03 22:28:40.968809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.968732 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f65bce2e-b44d-46cc-8e78-b7de172f8611" containerName="revoke-keys" Nov 03 22:28:40.968809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.968739 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65bce2e-b44d-46cc-8e78-b7de172f8611" containerName="revoke-keys" Nov 03 22:28:40.968905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.968897 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27a4e00-abaf-47bb-a3d3-6f2f59035d1c" containerName="revoke-keys" Nov 03 22:28:40.968937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.968911 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65bce2e-b44d-46cc-8e78-b7de172f8611" containerName="revoke-keys" Nov 03 22:28:40.972693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.972674 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:40.988450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:40.988425 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2"] Nov 03 22:28:41.030407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.030376 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" Nov 03 22:28:41.030798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.030408 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d" event={"ID":"f27a4e00-abaf-47bb-a3d3-6f2f59035d1c","Type":"ContainerDied","Data":"e240e2b316e19b1329a49220ddb48b8ee8f5c06a7a385a31b8bfc13d403af6f4"} Nov 03 22:28:41.030798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.030455 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e240e2b316e19b1329a49220ddb48b8ee8f5c06a7a385a31b8bfc13d403af6f4" Nov 03 22:28:41.097262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.097226 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076157af-jc5k2\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.097443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.097426 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh45x\" (UniqueName: \"kubernetes.io/projected/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-kube-api-access-gh45x\") pod \"maas-api-revoke-sub-076157af-jc5k2\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.097530 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.097450 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076157af-jc5k2\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.198872 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.198833 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076157af-jc5k2\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.199057 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.198990 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh45x\" (UniqueName: \"kubernetes.io/projected/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-kube-api-access-gh45x\") pod \"maas-api-revoke-sub-076157af-jc5k2\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.199057 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.199009 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076157af-jc5k2\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.199673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.199652 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076157af-jc5k2\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.201217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.201199 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076157af-jc5k2\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.220525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.220456 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh45x\" (UniqueName: \"kubernetes.io/projected/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-kube-api-access-gh45x\") pod \"maas-api-revoke-sub-076157af-jc5k2\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.288407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.288378 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:41.590340 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:41.590272 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe4ad9f_4eb1_486d_8de3_9024890fde1a.slice/crio-fdfe08335ac27efc6d377b5d96f313878954447f98e186f9b52bafb6ba35cb4a WatchSource:0}: Error finding container fdfe08335ac27efc6d377b5d96f313878954447f98e186f9b52bafb6ba35cb4a: Status 404 returned error can't find the container with id fdfe08335ac27efc6d377b5d96f313878954447f98e186f9b52bafb6ba35cb4a Nov 03 22:28:41.591751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:41.591727 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2"] Nov 03 22:28:42.039747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:42.039704 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" event={"ID":"bbe4ad9f-4eb1-486d-8de3-9024890fde1a","Type":"ContainerStarted","Data":"f0d28b5b5cd39021ab83c964195178f03f3401a5ec76fd3bacba255b5cd0e1a1"} Nov 03 22:28:42.040120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:42.039765 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" event={"ID":"bbe4ad9f-4eb1-486d-8de3-9024890fde1a","Type":"ContainerStarted","Data":"fdfe08335ac27efc6d377b5d96f313878954447f98e186f9b52bafb6ba35cb4a"} Nov 03 22:28:42.079311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:42.079245 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" podStartSLOduration=2.079231635 podStartE2EDuration="2.079231635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:42.075696049 +0000 UTC m=+948.479637026" watchObservedRunningTime="2026-11-03 22:28:42.079231635 +0000 UTC m=+948.483172610" Nov 03 22:28:43.049289 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.049255 4903 generic.go:356] "Generic (PLEG): container finished" podID="bbe4ad9f-4eb1-486d-8de3-9024890fde1a" containerID="f0d28b5b5cd39021ab83c964195178f03f3401a5ec76fd3bacba255b5cd0e1a1" exitCode=0 Nov 03 22:28:43.049747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.049324 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" event={"ID":"bbe4ad9f-4eb1-486d-8de3-9024890fde1a","Type":"ContainerDied","Data":"f0d28b5b5cd39021ab83c964195178f03f3401a5ec76fd3bacba255b5cd0e1a1"} Nov 03 22:28:43.566700 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.566663 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84"] Nov 03 22:28:43.573800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.573768 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.590869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.590839 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84"] Nov 03 22:28:43.631222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.631184 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.631438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.631330 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.631517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.631445 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.631517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.631499 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.631620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.631534 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.631620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.631559 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.631725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.631633 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.631725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.631697 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.631825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.631762 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6tw2\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-kube-api-access-z6tw2\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.733370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733340 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.733547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733380 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.733547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733399 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.733547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733450 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.733547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733500 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.734390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733554 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6tw2\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-kube-api-access-z6tw2\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.734390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733834 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.734390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733883 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.734390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733949 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.734390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.733987 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.734390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.734198 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.734390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.734249 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.734795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.734480 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.734795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.734624 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.736089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.736067 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.736435 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.736418 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.756471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.756443 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.756572 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.756471 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6tw2\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-kube-api-access-z6tw2\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:43.890659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:43.890378 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:44.150865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.150835 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq"] Nov 03 22:28:44.157000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.156974 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.195093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.195061 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq"] Nov 03 22:28:44.227167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.227138 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:44.240078 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:44.240053 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f7c6c6_3025_4adc_9695_0fbfb2ba017e.slice/crio-8492cf1101ef918ebb52f2d652250c71e6b00a295579b05fb5e9765402ac0203 WatchSource:0}: Error finding container 8492cf1101ef918ebb52f2d652250c71e6b00a295579b05fb5e9765402ac0203: Status 404 returned error can't find the container with id 8492cf1101ef918ebb52f2d652250c71e6b00a295579b05fb5e9765402ac0203 Nov 03 22:28:44.240940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.240919 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84"] Nov 03 22:28:44.241720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.241614 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzgbx\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-kube-api-access-xzgbx\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.241792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.241768 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.241849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.241812 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e059a1-1e9b-4a02-a988-b0744036103e-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.241945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.241927 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.242040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.242015 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e059a1-1e9b-4a02-a988-b0744036103e-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.242250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.242231 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.242349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.242264 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.242422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.242404 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.242477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.242463 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.242602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.242575 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:44.242641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.242634 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:44.242671 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.242663 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:44.344449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.344424 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-openshift-ca\") pod \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " Nov 03 22:28:44.345450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.344564 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-xks-ca\") pod \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " Nov 03 22:28:44.345450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.344681 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-kube-api-access-gh45x\" (UniqueName: \"kubernetes.io/projected/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-kube-api-access-gh45x\") pod \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\" (UID: \"bbe4ad9f-4eb1-486d-8de3-9024890fde1a\") " Nov 03 22:28:44.345450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.345018 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e059a1-1e9b-4a02-a988-b0744036103e-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.345450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.345078 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.345450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.345123 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.345450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.345214 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.345450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.345265 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.345450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.345322 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzgbx\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-kube-api-access-xzgbx\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.346220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.345488 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.346220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.345516 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e059a1-1e9b-4a02-a988-b0744036103e-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.346220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.345581 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.346406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.346221 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-openshift-ca" pod "bbe4ad9f-4eb1-486d-8de3-9024890fde1a" (UID: "bbe4ad9f-4eb1-486d-8de3-9024890fde1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:44.347051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.346833 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.347051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.346853 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-credential-socket\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.347195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.347139 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-certs\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.347547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.347525 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e059a1-1e9b-4a02-a988-b0744036103e-istiod-ca-cert\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.347916 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.347893 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-data\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.351895 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.351835 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-kube-api-access-gh45x" pod "bbe4ad9f-4eb1-486d-8de3-9024890fde1a" (UID: "bbe4ad9f-4eb1-486d-8de3-9024890fde1a"). InnerVolumeSpecName "kube-api-access-gh45x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:44.352192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.352150 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-envoy\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.353074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.353028 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e059a1-1e9b-4a02-a988-b0744036103e-istio-podinfo\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.356876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.356790 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-xks-ca" pod "bbe4ad9f-4eb1-486d-8de3-9024890fde1a" (UID: "bbe4ad9f-4eb1-486d-8de3-9024890fde1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:44.367126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.367102 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-istio-token\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.367411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.367393 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzgbx\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-kube-api-access-xzgbx\") pod \"e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.447148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.447072 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:44.447148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.447101 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh45x\" (UniqueName: \"kubernetes.io/projected/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-kube-api-access-gh45x\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:44.447148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.447111 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe4ad9f-4eb1-486d-8de3-9024890fde1a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:44.480702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.480668 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:44.809906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.809871 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq"] Nov 03 22:28:44.810791 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:44.810763 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e059a1_1e9b_4a02_a988_b0744036103e.slice/crio-0d8166336d8125b4a76920c5f35e7aea537ba95965338c316584a7dceab5c550 WatchSource:0}: Error finding container 0d8166336d8125b4a76920c5f35e7aea537ba95965338c316584a7dceab5c550: Status 404 returned error can't find the container with id 0d8166336d8125b4a76920c5f35e7aea537ba95965338c316584a7dceab5c550 Nov 03 22:28:44.813769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.813737 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:44.813877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.813799 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:44.813877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:44.813829 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:45.077588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.077489 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" event={"ID":"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e","Type":"ContainerStarted","Data":"ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334"} Nov 03 22:28:45.077588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.077552 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" event={"ID":"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e","Type":"ContainerStarted","Data":"8492cf1101ef918ebb52f2d652250c71e6b00a295579b05fb5e9765402ac0203"} Nov 03 22:28:45.079532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.079500 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" Nov 03 22:28:45.079532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.079520 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2" event={"ID":"bbe4ad9f-4eb1-486d-8de3-9024890fde1a","Type":"ContainerDied","Data":"fdfe08335ac27efc6d377b5d96f313878954447f98e186f9b52bafb6ba35cb4a"} Nov 03 22:28:45.079702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.079561 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdfe08335ac27efc6d377b5d96f313878954447f98e186f9b52bafb6ba35cb4a" Nov 03 22:28:45.082097 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.082069 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" event={"ID":"04e059a1-1e9b-4a02-a988-b0744036103e","Type":"ContainerStarted","Data":"f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1"} Nov 03 22:28:45.082233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.082120 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" event={"ID":"04e059a1-1e9b-4a02-a988-b0744036103e","Type":"ContainerStarted","Data":"0d8166336d8125b4a76920c5f35e7aea537ba95965338c316584a7dceab5c550"} Nov 03 22:28:45.130100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.130039 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" podStartSLOduration=2.13002504 podStartE2EDuration="2.13002504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:45.124758223 +0000 UTC m=+951.528699200" watchObservedRunningTime="2026-11-03 22:28:45.13002504 +0000 UTC m=+951.533966015" Nov 03 22:28:45.176321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.176253 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" podStartSLOduration=1.1762317 podStartE2EDuration="1.1762317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:45.164867019 +0000 UTC m=+951.568807997" watchObservedRunningTime="2026-11-03 22:28:45.1762317 +0000 UTC m=+951.580172674" Nov 03 22:28:45.488316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.486686 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:45.491783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.491420 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:28:45.491783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.491496 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:28:45.893452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.891157 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:45.893452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.892745 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:28:45.893452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:45.892804 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:28:46.481852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:46.481752 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:28:46.482485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:46.482448 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:28:46.899147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:46.898610 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:28:46.899147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:46.898683 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:28:47.296188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.296146 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv"] Nov 03 22:28:47.296997 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.296928 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bbe4ad9f-4eb1-486d-8de3-9024890fde1a" containerName="revoke-keys" Nov 03 22:28:47.296997 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.296954 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe4ad9f-4eb1-486d-8de3-9024890fde1a" containerName="revoke-keys" Nov 03 22:28:47.297180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.297101 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe4ad9f-4eb1-486d-8de3-9024890fde1a" containerName="revoke-keys" Nov 03 22:28:47.301340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.301320 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.312358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.312334 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv"] Nov 03 22:28:47.315450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.315417 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:47.315569 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.315468 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:28:47.327627 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.327507 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:28:47.327627 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.327546 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:28:47.328204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.328175 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f9feb45-xgnwv\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.328333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.328233 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f9feb45-xgnwv\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.328531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.328508 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smbhd\" (UniqueName: \"kubernetes.io/projected/bfe72668-863a-4438-8eb2-128814508bbe-kube-api-access-smbhd\") pod \"maas-api-revoke-sub-7f9feb45-xgnwv\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.429334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.429304 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f9feb45-xgnwv\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.429507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.429468 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smbhd\" (UniqueName: \"kubernetes.io/projected/bfe72668-863a-4438-8eb2-128814508bbe-kube-api-access-smbhd\") pod \"maas-api-revoke-sub-7f9feb45-xgnwv\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.429595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.429580 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f9feb45-xgnwv\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.430427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.430399 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f9feb45-xgnwv\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.432035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.432015 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f9feb45-xgnwv\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.497313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.496819 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smbhd\" (UniqueName: \"kubernetes.io/projected/bfe72668-863a-4438-8eb2-128814508bbe-kube-api-access-smbhd\") pod \"maas-api-revoke-sub-7f9feb45-xgnwv\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.497809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.497360 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:28:47.497809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.497427 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:28:47.618039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.617949 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:47.626723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.626691 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx"] Nov 03 22:28:47.633098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.632776 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:47.724254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.720088 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx"] Nov 03 22:28:47.733954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.733712 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrlwr\" (UniqueName: \"kubernetes.io/projected/df0188a6-db35-49b7-a3ae-75747cf438cb-kube-api-access-jrlwr\") pod \"maas-api-revoke-sub-3f20189f-vq4gx\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:47.733954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.733888 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f20189f-vq4gx\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:47.734498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.734250 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f20189f-vq4gx\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:47.836206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.835925 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f20189f-vq4gx\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:47.836481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.836370 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrlwr\" (UniqueName: \"kubernetes.io/projected/df0188a6-db35-49b7-a3ae-75747cf438cb-kube-api-access-jrlwr\") pod \"maas-api-revoke-sub-3f20189f-vq4gx\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:47.836481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.836433 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f20189f-vq4gx\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:47.837254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.837213 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f20189f-vq4gx\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:47.889380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.839547 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f20189f-vq4gx\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:47.902315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.900845 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:28:47.902315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.900894 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:28:47.935831 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:47.935803 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrlwr\" (UniqueName: \"kubernetes.io/projected/df0188a6-db35-49b7-a3ae-75747cf438cb-kube-api-access-jrlwr\") pod \"maas-api-revoke-sub-3f20189f-vq4gx\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:48.031381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:48.031353 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:48.136683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:48.135996 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv"] Nov 03 22:28:48.424124 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:28:48.424079 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:28:48.489041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:48.488475 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:28:48.489041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:48.488540 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:28:48.595248 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:48.595213 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0188a6_db35_49b7_a3ae_75747cf438cb.slice/crio-5385ef564d041a13c0efebe8f304fef93e4ddba007be4074906a54e76de3a5a6 WatchSource:0}: Error finding container 5385ef564d041a13c0efebe8f304fef93e4ddba007be4074906a54e76de3a5a6: Status 404 returned error can't find the container with id 5385ef564d041a13c0efebe8f304fef93e4ddba007be4074906a54e76de3a5a6 Nov 03 22:28:48.595664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:48.595565 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx"] Nov 03 22:28:48.894367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:48.893987 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:28:48.894367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:48.894045 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:28:49.219377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.219337 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" event={"ID":"df0188a6-db35-49b7-a3ae-75747cf438cb","Type":"ContainerStarted","Data":"597950f6499dbf4c0d259ef189d05b789d4b5f84594322301c2b41044af00fb6"} Nov 03 22:28:49.219694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.219405 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" event={"ID":"df0188a6-db35-49b7-a3ae-75747cf438cb","Type":"ContainerStarted","Data":"5385ef564d041a13c0efebe8f304fef93e4ddba007be4074906a54e76de3a5a6"} Nov 03 22:28:49.221989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.221961 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" event={"ID":"bfe72668-863a-4438-8eb2-128814508bbe","Type":"ContainerStarted","Data":"ac2213b8db97a79f4955b4e70c02765947d47f4ed2bccf00987302214c187e8b"} Nov 03 22:28:49.222094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.222005 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" event={"ID":"bfe72668-863a-4438-8eb2-128814508bbe","Type":"ContainerStarted","Data":"86285fde57f171bb68b06a8c6bee15dbe70f8ff9cdf291ac42662567d6f99371"} Nov 03 22:28:49.327448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.327392 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" podStartSLOduration=2.3273775 podStartE2EDuration="2.3273775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:49.325040539 +0000 UTC m=+955.728981514" watchObservedRunningTime="2026-11-03 22:28:49.3273775 +0000 UTC m=+955.731318475" Nov 03 22:28:49.329961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.329928 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" podStartSLOduration=2.329919553 podStartE2EDuration="2.329919553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:49.284059335 +0000 UTC m=+955.688000310" watchObservedRunningTime="2026-11-03 22:28:49.329919553 +0000 UTC m=+955.733860528" Nov 03 22:28:49.486324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.482826 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:28:49.486324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.482901 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:28:49.915193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.915156 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:28:49.915756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:49.915400 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:28:50.234074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:50.233973 4903 generic.go:356] "Generic (PLEG): container finished" podID="df0188a6-db35-49b7-a3ae-75747cf438cb" containerID="597950f6499dbf4c0d259ef189d05b789d4b5f84594322301c2b41044af00fb6" exitCode=0 Nov 03 22:28:50.234074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:50.234049 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" event={"ID":"df0188a6-db35-49b7-a3ae-75747cf438cb","Type":"ContainerDied","Data":"597950f6499dbf4c0d259ef189d05b789d4b5f84594322301c2b41044af00fb6"} Nov 03 22:28:50.235591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:50.235556 4903 generic.go:356] "Generic (PLEG): container finished" podID="bfe72668-863a-4438-8eb2-128814508bbe" containerID="ac2213b8db97a79f4955b4e70c02765947d47f4ed2bccf00987302214c187e8b" exitCode=0 Nov 03 22:28:50.235654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:50.235635 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" event={"ID":"bfe72668-863a-4438-8eb2-128814508bbe","Type":"ContainerDied","Data":"ac2213b8db97a79f4955b4e70c02765947d47f4ed2bccf00987302214c187e8b"} Nov 03 22:28:50.481754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:50.481718 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:28:50.481948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:50.481768 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:28:50.891345 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:50.891278 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 22:28:50.891793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:50.891730 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 22:28:51.482312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.480995 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 22:28:51.482312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.481046 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 22:28:51.530082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.530014 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:51.534342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.534279 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:51.552371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.552307 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe72668-863a-4438-8eb2-128814508bbe-kube-api-access-smbhd\" (UniqueName: \"kubernetes.io/projected/bfe72668-863a-4438-8eb2-128814508bbe-kube-api-access-smbhd\") pod \"bfe72668-863a-4438-8eb2-128814508bbe\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " Nov 03 22:28:51.552567 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.552375 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-openshift-ca\") pod \"df0188a6-db35-49b7-a3ae-75747cf438cb\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " Nov 03 22:28:51.552567 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.552479 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-xks-ca\") pod \"df0188a6-db35-49b7-a3ae-75747cf438cb\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " Nov 03 22:28:51.552567 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.552535 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-openshift-ca\") pod \"bfe72668-863a-4438-8eb2-128814508bbe\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " Nov 03 22:28:51.552766 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.552726 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-xks-ca\") pod \"bfe72668-863a-4438-8eb2-128814508bbe\" (UID: \"bfe72668-863a-4438-8eb2-128814508bbe\") " Nov 03 22:28:51.552829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.552786 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0188a6-db35-49b7-a3ae-75747cf438cb-kube-api-access-jrlwr\" (UniqueName: \"kubernetes.io/projected/df0188a6-db35-49b7-a3ae-75747cf438cb-kube-api-access-jrlwr\") pod \"df0188a6-db35-49b7-a3ae-75747cf438cb\" (UID: \"df0188a6-db35-49b7-a3ae-75747cf438cb\") " Nov 03 22:28:51.553352 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.553321 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-openshift-ca" pod "bfe72668-863a-4438-8eb2-128814508bbe" (UID: "bfe72668-863a-4438-8eb2-128814508bbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:51.553630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.553607 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:51.553740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.553681 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-openshift-ca" pod "df0188a6-db35-49b7-a3ae-75747cf438cb" (UID: "df0188a6-db35-49b7-a3ae-75747cf438cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:51.555011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.554982 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe72668-863a-4438-8eb2-128814508bbe-kube-api-access-smbhd" pod "bfe72668-863a-4438-8eb2-128814508bbe" (UID: "bfe72668-863a-4438-8eb2-128814508bbe"). InnerVolumeSpecName "kube-api-access-smbhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:51.556927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.556897 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-xks-ca" pod "bfe72668-863a-4438-8eb2-128814508bbe" (UID: "bfe72668-863a-4438-8eb2-128814508bbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:51.557051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.556910 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-xks-ca" pod "df0188a6-db35-49b7-a3ae-75747cf438cb" (UID: "df0188a6-db35-49b7-a3ae-75747cf438cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:51.557546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.557516 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0188a6-db35-49b7-a3ae-75747cf438cb-kube-api-access-jrlwr" pod "df0188a6-db35-49b7-a3ae-75747cf438cb" (UID: "df0188a6-db35-49b7-a3ae-75747cf438cb"). InnerVolumeSpecName "kube-api-access-jrlwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:51.655334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.655202 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smbhd\" (UniqueName: \"kubernetes.io/projected/bfe72668-863a-4438-8eb2-128814508bbe-kube-api-access-smbhd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:51.655334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.655241 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:51.655334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.655258 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0188a6-db35-49b7-a3ae-75747cf438cb-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:51.655334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.655274 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe72668-863a-4438-8eb2-128814508bbe-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:51.655334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:51.655288 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrlwr\" (UniqueName: \"kubernetes.io/projected/df0188a6-db35-49b7-a3ae-75747cf438cb-kube-api-access-jrlwr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:52.016990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.016951 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:52.017662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.017636 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:52.020343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.018923 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:28:52.264079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.264045 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" Nov 03 22:28:52.264276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.264051 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx" event={"ID":"df0188a6-db35-49b7-a3ae-75747cf438cb","Type":"ContainerDied","Data":"5385ef564d041a13c0efebe8f304fef93e4ddba007be4074906a54e76de3a5a6"} Nov 03 22:28:52.264276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.264201 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5385ef564d041a13c0efebe8f304fef93e4ddba007be4074906a54e76de3a5a6" Nov 03 22:28:52.265628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.265608 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" Nov 03 22:28:52.265743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.265638 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv" event={"ID":"bfe72668-863a-4438-8eb2-128814508bbe","Type":"ContainerDied","Data":"86285fde57f171bb68b06a8c6bee15dbe70f8ff9cdf291ac42662567d6f99371"} Nov 03 22:28:52.265743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.265684 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86285fde57f171bb68b06a8c6bee15dbe70f8ff9cdf291ac42662567d6f99371" Nov 03 22:28:52.579489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.579400 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:52.580018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.579915 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:52.580778 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:52.580758 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:28:54.374955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.374921 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs"] Nov 03 22:28:54.375677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.375661 4903 cpu_manager.go:414] "containerMap: removing container" podUID="df0188a6-db35-49b7-a3ae-75747cf438cb" containerName="revoke-keys" Nov 03 22:28:54.375677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.375678 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0188a6-db35-49b7-a3ae-75747cf438cb" containerName="revoke-keys" Nov 03 22:28:54.375767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.375694 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bfe72668-863a-4438-8eb2-128814508bbe" containerName="revoke-keys" Nov 03 22:28:54.375767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.375701 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe72668-863a-4438-8eb2-128814508bbe" containerName="revoke-keys" Nov 03 22:28:54.375837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.375827 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0188a6-db35-49b7-a3ae-75747cf438cb" containerName="revoke-keys" Nov 03 22:28:54.375881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.375839 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe72668-863a-4438-8eb2-128814508bbe" containerName="revoke-keys" Nov 03 22:28:54.381349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.381329 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.394709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.394686 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:28:54.405145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.405114 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:28:54.405145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.405100 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:28:54.405145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.405152 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:28:54.412910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.412885 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs"] Nov 03 22:28:54.421715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.421682 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bde9e295-rb9cs\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.421878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.421747 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwpxg\" (UniqueName: \"kubernetes.io/projected/788ad52d-efde-4a24-bb83-63453db6e0a0-kube-api-access-gwpxg\") pod \"maas-api-revoke-sub-bde9e295-rb9cs\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.421947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.421883 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bde9e295-rb9cs\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.523742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.523700 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bde9e295-rb9cs\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.523975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.523957 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bde9e295-rb9cs\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.524038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.524005 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwpxg\" (UniqueName: \"kubernetes.io/projected/788ad52d-efde-4a24-bb83-63453db6e0a0-kube-api-access-gwpxg\") pod \"maas-api-revoke-sub-bde9e295-rb9cs\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.524678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.524656 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bde9e295-rb9cs\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.526272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.526247 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bde9e295-rb9cs\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.545388 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.545354 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwpxg\" (UniqueName: \"kubernetes.io/projected/788ad52d-efde-4a24-bb83-63453db6e0a0-kube-api-access-gwpxg\") pod \"maas-api-revoke-sub-bde9e295-rb9cs\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:54.700183 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:54.700140 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:55.009056 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:55.009027 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788ad52d_efde_4a24_bb83_63453db6e0a0.slice/crio-f9a8a04fc59312f6ad5ae6a4edbca2fa0621abf0b335f9b596bd2bf9f36db117 WatchSource:0}: Error finding container f9a8a04fc59312f6ad5ae6a4edbca2fa0621abf0b335f9b596bd2bf9f36db117: Status 404 returned error can't find the container with id f9a8a04fc59312f6ad5ae6a4edbca2fa0621abf0b335f9b596bd2bf9f36db117 Nov 03 22:28:55.010213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.010184 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs"] Nov 03 22:28:55.294085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.294048 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" event={"ID":"788ad52d-efde-4a24-bb83-63453db6e0a0","Type":"ContainerStarted","Data":"f9a8a04fc59312f6ad5ae6a4edbca2fa0621abf0b335f9b596bd2bf9f36db117"} Nov 03 22:28:55.640668 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.640525 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps"] Nov 03 22:28:55.645654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.645622 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.664722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.664684 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps"] Nov 03 22:28:55.741640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.741403 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ff0633-qmlps\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.741640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.741603 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wpz8\" (UniqueName: \"kubernetes.io/projected/bdb3ef8d-6902-45fb-a727-25c1737cf73f-kube-api-access-9wpz8\") pod \"maas-api-revoke-sub-c8ff0633-qmlps\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.742056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.741760 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ff0633-qmlps\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.842636 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.842599 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ff0633-qmlps\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.842818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.842671 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ff0633-qmlps\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.842887 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.842856 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wpz8\" (UniqueName: \"kubernetes.io/projected/bdb3ef8d-6902-45fb-a727-25c1737cf73f-kube-api-access-9wpz8\") pod \"maas-api-revoke-sub-c8ff0633-qmlps\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.843369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.843344 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8ff0633-qmlps\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.845190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.845166 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8ff0633-qmlps\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.865902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.865878 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wpz8\" (UniqueName: \"kubernetes.io/projected/bdb3ef8d-6902-45fb-a727-25c1737cf73f-kube-api-access-9wpz8\") pod \"maas-api-revoke-sub-c8ff0633-qmlps\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:55.965587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:55.965558 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:56.307276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:56.307185 4903 generic.go:356] "Generic (PLEG): container finished" podID="788ad52d-efde-4a24-bb83-63453db6e0a0" containerID="022f79b5250c9f870808a19aa86f0156cce083242453f7cad9a5843d115830de" exitCode=0 Nov 03 22:28:56.307470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:56.307269 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" event={"ID":"788ad52d-efde-4a24-bb83-63453db6e0a0","Type":"ContainerDied","Data":"022f79b5250c9f870808a19aa86f0156cce083242453f7cad9a5843d115830de"} Nov 03 22:28:56.337196 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:56.337164 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb3ef8d_6902_45fb_a727_25c1737cf73f.slice/crio-aac77e61cd1ec9047433f4000a84846361d84ef599c44ff06a3b2ced6e7ee9e2 WatchSource:0}: Error finding container aac77e61cd1ec9047433f4000a84846361d84ef599c44ff06a3b2ced6e7ee9e2: Status 404 returned error can't find the container with id aac77e61cd1ec9047433f4000a84846361d84ef599c44ff06a3b2ced6e7ee9e2 Nov 03 22:28:56.338642 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:56.338618 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps"] Nov 03 22:28:57.325805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.325760 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" event={"ID":"bdb3ef8d-6902-45fb-a727-25c1737cf73f","Type":"ContainerDied","Data":"c690009d05d9879ea3f473199fc32bd0078a91bfd78ab9b8c9dfa5217996e323"} Nov 03 22:28:57.326227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.325919 4903 generic.go:356] "Generic (PLEG): container finished" podID="bdb3ef8d-6902-45fb-a727-25c1737cf73f" containerID="c690009d05d9879ea3f473199fc32bd0078a91bfd78ab9b8c9dfa5217996e323" exitCode=0 Nov 03 22:28:57.326227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.326190 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" event={"ID":"bdb3ef8d-6902-45fb-a727-25c1737cf73f","Type":"ContainerStarted","Data":"aac77e61cd1ec9047433f4000a84846361d84ef599c44ff06a3b2ced6e7ee9e2"} Nov 03 22:28:57.554769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.554741 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:57.569728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.569651 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788ad52d-efde-4a24-bb83-63453db6e0a0-kube-api-access-gwpxg\" (UniqueName: \"kubernetes.io/projected/788ad52d-efde-4a24-bb83-63453db6e0a0-kube-api-access-gwpxg\") pod \"788ad52d-efde-4a24-bb83-63453db6e0a0\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " Nov 03 22:28:57.569879 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.569829 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-xks-ca\") pod \"788ad52d-efde-4a24-bb83-63453db6e0a0\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " Nov 03 22:28:57.569929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.569920 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-openshift-ca\") pod \"788ad52d-efde-4a24-bb83-63453db6e0a0\" (UID: \"788ad52d-efde-4a24-bb83-63453db6e0a0\") " Nov 03 22:28:57.570409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.570384 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-openshift-ca" pod "788ad52d-efde-4a24-bb83-63453db6e0a0" (UID: "788ad52d-efde-4a24-bb83-63453db6e0a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:57.570552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.570540 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:57.572350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.572320 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-xks-ca" pod "788ad52d-efde-4a24-bb83-63453db6e0a0" (UID: "788ad52d-efde-4a24-bb83-63453db6e0a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:57.572470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.572352 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788ad52d-efde-4a24-bb83-63453db6e0a0-kube-api-access-gwpxg" pod "788ad52d-efde-4a24-bb83-63453db6e0a0" (UID: "788ad52d-efde-4a24-bb83-63453db6e0a0"). InnerVolumeSpecName "kube-api-access-gwpxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:57.671913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.671876 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwpxg\" (UniqueName: \"kubernetes.io/projected/788ad52d-efde-4a24-bb83-63453db6e0a0-kube-api-access-gwpxg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:57.671913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:57.671907 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788ad52d-efde-4a24-bb83-63453db6e0a0-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:58.336403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.336368 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" Nov 03 22:28:58.336403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.336398 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs" event={"ID":"788ad52d-efde-4a24-bb83-63453db6e0a0","Type":"ContainerDied","Data":"f9a8a04fc59312f6ad5ae6a4edbca2fa0621abf0b335f9b596bd2bf9f36db117"} Nov 03 22:28:58.336827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.336459 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9a8a04fc59312f6ad5ae6a4edbca2fa0621abf0b335f9b596bd2bf9f36db117" Nov 03 22:28:58.411399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.404639 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4"] Nov 03 22:28:58.411399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.405696 4903 cpu_manager.go:414] "containerMap: removing container" podUID="788ad52d-efde-4a24-bb83-63453db6e0a0" containerName="revoke-keys" Nov 03 22:28:58.411399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.405717 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788ad52d-efde-4a24-bb83-63453db6e0a0" containerName="revoke-keys" Nov 03 22:28:58.411399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.405931 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="788ad52d-efde-4a24-bb83-63453db6e0a0" containerName="revoke-keys" Nov 03 22:28:58.411399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.410647 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.420651 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.420622 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz"] Nov 03 22:28:58.428627 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.427060 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.431595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.431563 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4"] Nov 03 22:28:58.451358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.451330 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz"] Nov 03 22:28:58.483667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.483628 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.483969 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.483943 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvzb\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-kube-api-access-hxvzb\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.484182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484159 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.484277 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484207 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.484383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484365 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.484425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484403 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.484601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484571 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74942a-efdf-4726-a427-efd8c125ec7e-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.484704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484632 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.484704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484686 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.484817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484723 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.484817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484775 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.484927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484815 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92980776-9c6e-48e9-ae02-4dbaecabc575-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.484927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484850 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.484927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484879 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.484927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484920 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pvkf\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-kube-api-access-4pvkf\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.485126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.484947 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.485126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.485009 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.485126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.485040 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.522497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.522470 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:58.586559 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.586462 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-xks-ca\") pod \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " Nov 03 22:28:58.586716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.586604 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-openshift-ca\") pod \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " Nov 03 22:28:58.586716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.586660 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdb3ef8d-6902-45fb-a727-25c1737cf73f-kube-api-access-9wpz8\" (UniqueName: \"kubernetes.io/projected/bdb3ef8d-6902-45fb-a727-25c1737cf73f-kube-api-access-9wpz8\") pod \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\" (UID: \"bdb3ef8d-6902-45fb-a727-25c1737cf73f\") " Nov 03 22:28:58.587202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587178 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.587340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587225 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.587340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587330 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.587469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587367 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.587469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587446 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74942a-efdf-4726-a427-efd8c125ec7e-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.587578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587478 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.587578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587533 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.587578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587560 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.587735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587593 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.587735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587621 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92980776-9c6e-48e9-ae02-4dbaecabc575-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.587735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587645 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.587735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587672 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.587735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587705 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pvkf\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-kube-api-access-4pvkf\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.587735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587722 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.588044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587765 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.588044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587795 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.588044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587836 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.588044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.587862 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvzb\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-kube-api-access-hxvzb\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.588375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.588344 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-openshift-ca" pod "bdb3ef8d-6902-45fb-a727-25c1737cf73f" (UID: "bdb3ef8d-6902-45fb-a727-25c1737cf73f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:28:58.588677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.588648 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.590280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.588963 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-credential-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.590280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.589081 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74942a-efdf-4726-a427-efd8c125ec7e-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.590280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.589407 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.590280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.589506 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.590280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.589917 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-xks-ca" pod "bdb3ef8d-6902-45fb-a727-25c1737cf73f" (UID: "bdb3ef8d-6902-45fb-a727-25c1737cf73f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:28:58.590280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.589983 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.593203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.590545 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-data\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.593203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.590724 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.593203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.590846 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb3ef8d-6902-45fb-a727-25c1737cf73f-kube-api-access-9wpz8" pod "bdb3ef8d-6902-45fb-a727-25c1737cf73f" (UID: "bdb3ef8d-6902-45fb-a727-25c1737cf73f"). InnerVolumeSpecName "kube-api-access-9wpz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:28:58.593203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.590975 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-certs\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.593203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.591161 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92980776-9c6e-48e9-ae02-4dbaecabc575-istiod-ca-cert\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.593203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.591228 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-socket\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.593203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.592793 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-envoy\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.593660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.593628 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.594487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.594458 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-podinfo\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.617852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.617826 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvzb\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-kube-api-access-hxvzb\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.618139 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.618115 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.618387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.618368 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-token\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.618458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.618396 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pvkf\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-kube-api-access-4pvkf\") pod \"e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:58.689751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.689718 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:58.689751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.689748 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wpz8\" (UniqueName: \"kubernetes.io/projected/bdb3ef8d-6902-45fb-a727-25c1737cf73f-kube-api-access-9wpz8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:58.689751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.689758 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdb3ef8d-6902-45fb-a727-25c1737cf73f-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:28:58.732946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.732916 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:58.756917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:58.756887 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:28:59.070492 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:59.070416 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d74942a_efdf_4726_a427_efd8c125ec7e.slice/crio-fe967024fe01f45a541b39b56bbaceed11d3fe5fe0502e347fb7c1cf356e03a4 WatchSource:0}: Error finding container fe967024fe01f45a541b39b56bbaceed11d3fe5fe0502e347fb7c1cf356e03a4: Status 404 returned error can't find the container with id fe967024fe01f45a541b39b56bbaceed11d3fe5fe0502e347fb7c1cf356e03a4 Nov 03 22:28:59.072129 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.072104 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4"] Nov 03 22:28:59.073481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.073445 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:59.073577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.073524 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:59.073640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.073575 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:59.094322 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:28:59.086996 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92980776_9c6e_48e9_ae02_4dbaecabc575.slice/crio-fa3a5917e262f8310e731c1811c9c9d0f73e24f08d7830dac86d4239d885ad2a WatchSource:0}: Error finding container fa3a5917e262f8310e731c1811c9c9d0f73e24f08d7830dac86d4239d885ad2a: Status 404 returned error can't find the container with id fa3a5917e262f8310e731c1811c9c9d0f73e24f08d7830dac86d4239d885ad2a Nov 03 22:28:59.094322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.088603 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz"] Nov 03 22:28:59.094322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.090228 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:59.094322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.090345 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:59.094322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.090392 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:28:59.305120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.305080 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26"] Nov 03 22:28:59.305882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.305859 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bdb3ef8d-6902-45fb-a727-25c1737cf73f" containerName="revoke-keys" Nov 03 22:28:59.305882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.305882 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdb3ef8d-6902-45fb-a727-25c1737cf73f" containerName="revoke-keys" Nov 03 22:28:59.306079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.306036 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdb3ef8d-6902-45fb-a727-25c1737cf73f" containerName="revoke-keys" Nov 03 22:28:59.310131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.310105 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.339922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.339889 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26"] Nov 03 22:28:59.355320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.355167 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" event={"ID":"bdb3ef8d-6902-45fb-a727-25c1737cf73f","Type":"ContainerDied","Data":"aac77e61cd1ec9047433f4000a84846361d84ef599c44ff06a3b2ced6e7ee9e2"} Nov 03 22:28:59.355320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.355226 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aac77e61cd1ec9047433f4000a84846361d84ef599c44ff06a3b2ced6e7ee9e2" Nov 03 22:28:59.355661 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.355636 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps" Nov 03 22:28:59.371094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.370334 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" event={"ID":"92980776-9c6e-48e9-ae02-4dbaecabc575","Type":"ContainerStarted","Data":"4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69"} Nov 03 22:28:59.371094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.370397 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" event={"ID":"92980776-9c6e-48e9-ae02-4dbaecabc575","Type":"ContainerStarted","Data":"fa3a5917e262f8310e731c1811c9c9d0f73e24f08d7830dac86d4239d885ad2a"} Nov 03 22:28:59.375243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.374431 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" event={"ID":"0d74942a-efdf-4726-a427-efd8c125ec7e","Type":"ContainerStarted","Data":"753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63"} Nov 03 22:28:59.375243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.374486 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" event={"ID":"0d74942a-efdf-4726-a427-efd8c125ec7e","Type":"ContainerStarted","Data":"fe967024fe01f45a541b39b56bbaceed11d3fe5fe0502e347fb7c1cf356e03a4"} Nov 03 22:28:59.406373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.405907 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.406373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.405963 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.406373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.406078 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.406373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.406141 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t4c4\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-kube-api-access-2t4c4\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.406373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.406183 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.406373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.406221 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.407519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.407495 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.407655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.407554 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.407655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.407604 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.416122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.416065 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" podStartSLOduration=1.416047193 podStartE2EDuration="1.416047193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:59.412245528 +0000 UTC m=+965.816186503" watchObservedRunningTime="2026-11-03 22:28:59.416047193 +0000 UTC m=+965.819988170" Nov 03 22:28:59.474003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.473938 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" podStartSLOduration=1.47391965 podStartE2EDuration="1.47391965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:28:59.469557019 +0000 UTC m=+965.873497988" watchObservedRunningTime="2026-11-03 22:28:59.47391965 +0000 UTC m=+965.877860626" Nov 03 22:28:59.509404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509367 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.509579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509412 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.509579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509478 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.509579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509511 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t4c4\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-kube-api-access-2t4c4\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.509579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509548 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.509818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509584 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.509818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509701 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.509818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509735 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.509818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509778 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.510076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.509967 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.510076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.510036 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-data\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.510184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.510128 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-certs\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.510281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.510260 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-credential-socket\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.510360 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.510327 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istiod-ca-cert\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.512096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.512067 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-envoy\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.512283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.512261 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-podinfo\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.531042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.531016 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-token\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.531147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.531023 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t4c4\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-kube-api-access-2t4c4\") pod \"e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.639737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.639646 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:28:59.739420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.739392 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:28:59.759075 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:28:59.759036 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:29:00.002210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.001247 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26"] Nov 03 22:29:00.008465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.008421 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:00.008626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.008504 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:00.008626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.008547 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:00.400977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.400936 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" event={"ID":"0a9c0fa0-1913-4627-b146-d4d4f8f2826b","Type":"ContainerStarted","Data":"f9fef8b2e5916de7aa705ffb82a2cecbab2e546f0779ff74d430a96bb80baa89"} Nov 03 22:29:00.402336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.402308 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" event={"ID":"0a9c0fa0-1913-4627-b146-d4d4f8f2826b","Type":"ContainerStarted","Data":"42148a39655917176fa94b8994c46a73e128b14b4b0718ca743522f20b1a4be2"} Nov 03 22:29:00.489317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.488407 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" podStartSLOduration=1.470269932 podStartE2EDuration="1.470269932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:00.462883143 +0000 UTC m=+966.866824156" watchObservedRunningTime="2026-11-03 22:29:00.470269932 +0000 UTC m=+966.874210909" Nov 03 22:29:00.641446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.641407 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:29:00.735120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.735039 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 22:29:00.735120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.735092 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 22:29:00.773316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.772591 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:29:00.773316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:00.772650 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 22:29:01.667330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:01.661452 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 22:29:01.722534 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:01.722492 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 22:29:01.733285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:01.733253 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 22:29:01.733552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:01.733526 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 22:29:01.836318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:01.824819 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:29:01.836318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:01.824883 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 22:29:01.884318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:01.883644 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t"] Nov 03 22:29:02.028410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.012521 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t"] Nov 03 22:29:02.028706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.020334 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:02.216332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.209654 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b279244\"" Nov 03 22:29:02.216332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.211165 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtfbz\" (UniqueName: \"kubernetes.io/projected/788c17f5-3df0-45eb-b35c-390eddb3beeb-kube-api-access-xtfbz\") pod \"payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t\" (UID: \"788c17f5-3df0-45eb-b35c-390eddb3beeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:02.216332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.211470 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788c17f5-3df0-45eb-b35c-390eddb3beeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t\" (UID: \"788c17f5-3df0-45eb-b35c-390eddb3beeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:02.216332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.212701 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1b279244-dockercfg-d2wmh\"" Nov 03 22:29:02.316125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.315210 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788c17f5-3df0-45eb-b35c-390eddb3beeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t\" (UID: \"788c17f5-3df0-45eb-b35c-390eddb3beeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:02.316125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.315338 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfbz\" (UniqueName: \"kubernetes.io/projected/788c17f5-3df0-45eb-b35c-390eddb3beeb-kube-api-access-xtfbz\") pod \"payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t\" (UID: \"788c17f5-3df0-45eb-b35c-390eddb3beeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:02.317111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.317003 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788c17f5-3df0-45eb-b35c-390eddb3beeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t\" (UID: \"788c17f5-3df0-45eb-b35c-390eddb3beeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:02.409318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.389119 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfbz\" (UniqueName: \"kubernetes.io/projected/788c17f5-3df0-45eb-b35c-390eddb3beeb-kube-api-access-xtfbz\") pod \"payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t\" (UID: \"788c17f5-3df0-45eb-b35c-390eddb3beeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:02.452318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.429246 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:02.733127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.733092 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 22:29:02.823085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.823043 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 22:29:02.823791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.802909 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:29:02.823995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.823970 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 22:29:02.824137 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.802954 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 22:29:02.824242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:02.824224 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 22:29:03.497834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:03.495905 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t"] Nov 03 22:29:03.660320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:03.640610 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 22:29:03.660320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:03.640668 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 22:29:03.696364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:03.691889 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" event={"ID":"788c17f5-3df0-45eb-b35c-390eddb3beeb","Type":"ContainerStarted","Data":"06113159bab69c4d1a7b414d59c796180df3257087f2b5b05e304791474dd773"} Nov 03 22:29:03.736428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:03.736391 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 22:29:03.736960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:03.736929 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 22:29:03.817319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:03.816114 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:29:03.817319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:03.816175 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 22:29:04.646410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.645451 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 22:29:04.646410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.645521 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 22:29:04.727033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.726991 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" event={"ID":"788c17f5-3df0-45eb-b35c-390eddb3beeb","Type":"ContainerStarted","Data":"80de6e25c0c1d7edad5ebe796dce3f85d478699179fffe7427995dd38a403868"} Nov 03 22:29:04.727330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.727284 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:04.733774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.733749 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 22:29:04.733952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.733932 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 22:29:04.766454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.765885 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt"] Nov 03 22:29:04.767597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.767490 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:29:04.767597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.767541 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 22:29:04.786541 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.786508 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt"] Nov 03 22:29:04.786827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.786813 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:04.826318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.802214 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:29:04.826944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.819089 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:29:04.827059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.819141 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:04.827059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.819192 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:29:04.836318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.828560 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2478f748-mhwkt\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:04.836318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.828723 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fj5\" (UniqueName: \"kubernetes.io/projected/00605c20-be93-44bc-8e74-c80d43371bf2-kube-api-access-29fj5\") pod \"maas-api-revoke-sub-2478f748-mhwkt\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:04.836318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.828861 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2478f748-mhwkt\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:04.878318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.840927 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" podStartSLOduration=3.840907326 podStartE2EDuration="3.840907326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:04.783586284 +0000 UTC m=+971.187527287" watchObservedRunningTime="2026-11-03 22:29:04.840907326 +0000 UTC m=+971.244848302" Nov 03 22:29:04.934105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.934052 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2478f748-mhwkt\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:04.934606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.934574 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fj5\" (UniqueName: \"kubernetes.io/projected/00605c20-be93-44bc-8e74-c80d43371bf2-kube-api-access-29fj5\") pod \"maas-api-revoke-sub-2478f748-mhwkt\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:04.934886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.934855 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2478f748-mhwkt\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:04.936038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:04.935968 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2478f748-mhwkt\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:05.131318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.124924 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fj5\" (UniqueName: \"kubernetes.io/projected/00605c20-be93-44bc-8e74-c80d43371bf2-kube-api-access-29fj5\") pod \"maas-api-revoke-sub-2478f748-mhwkt\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:05.145193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.145161 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2478f748-mhwkt\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:05.221370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.220538 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:05.647323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.642678 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 22:29:05.647323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.642739 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 22:29:05.733429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.733369 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 22:29:05.733429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.733423 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 22:29:05.787644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.786930 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 22:29:05.795376 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.795198 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 22:29:05.832413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:05.832384 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt"] Nov 03 22:29:05.832920 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:29:05.832813 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00605c20_be93_44bc_8e74_c80d43371bf2.slice/crio-7a58496ca9e26d8fc0cf6a605a7604996e51b44c5af4ada2c55a0bfb622c028c WatchSource:0}: Error finding container 7a58496ca9e26d8fc0cf6a605a7604996e51b44c5af4ada2c55a0bfb622c028c: Status 404 returned error can't find the container with id 7a58496ca9e26d8fc0cf6a605a7604996e51b44c5af4ada2c55a0bfb622c028c Nov 03 22:29:06.640405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:06.640243 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 22:29:06.640405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:06.640337 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 22:29:06.761640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:06.761568 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" event={"ID":"00605c20-be93-44bc-8e74-c80d43371bf2","Type":"ContainerStarted","Data":"04b3620d9b8aec6eaa20573742fdcaf8be21264f810ee527faeb7b11a185a668"} Nov 03 22:29:06.761640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:06.761639 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" event={"ID":"00605c20-be93-44bc-8e74-c80d43371bf2","Type":"ContainerStarted","Data":"7a58496ca9e26d8fc0cf6a605a7604996e51b44c5af4ada2c55a0bfb622c028c"} Nov 03 22:29:06.808818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:06.808342 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" podStartSLOduration=2.808323167 podStartE2EDuration="2.808323167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:06.799008558 +0000 UTC m=+973.202949534" watchObservedRunningTime="2026-11-03 22:29:06.808323167 +0000 UTC m=+973.212264146" Nov 03 22:29:07.311683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.311651 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:29:07.314694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.314672 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:29:07.317837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.317820 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:29:07.431318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.425348 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:29:07.431318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.426836 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:29:07.431318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.429106 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:29:07.646729 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.646692 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 22:29:07.646880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.646763 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 22:29:07.814911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.807331 4903 generic.go:356] "Generic (PLEG): container finished" podID="00605c20-be93-44bc-8e74-c80d43371bf2" containerID="04b3620d9b8aec6eaa20573742fdcaf8be21264f810ee527faeb7b11a185a668" exitCode=0 Nov 03 22:29:07.814911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:07.808980 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" event={"ID":"00605c20-be93-44bc-8e74-c80d43371bf2","Type":"ContainerDied","Data":"04b3620d9b8aec6eaa20573742fdcaf8be21264f810ee527faeb7b11a185a668"} Nov 03 22:29:08.659901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:08.659862 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:29:08.661442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:08.661418 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:29:08.664010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:08.663984 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:29:09.413017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.410535 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9"] Nov 03 22:29:09.431318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.424051 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:09.551317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.528596 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzvg\" (UniqueName: \"kubernetes.io/projected/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-kube-api-access-2fzvg\") pod \"maas-api-revoke-sub-b48b994d-qztv9\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:09.551317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.528681 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b48b994d-qztv9\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:09.551317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.528779 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b48b994d-qztv9\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:09.663384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.640789 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzvg\" (UniqueName: \"kubernetes.io/projected/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-kube-api-access-2fzvg\") pod \"maas-api-revoke-sub-b48b994d-qztv9\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:09.698145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.696551 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b48b994d-qztv9\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:09.698145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.696745 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b48b994d-qztv9\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:09.698145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.674366 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9"] Nov 03 22:29:09.698145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.698091 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b48b994d-qztv9\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:09.782271 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.780797 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b48b994d-qztv9\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:09.920639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.903549 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 03 22:29:09.973224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:09.968031 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzvg\" (UniqueName: \"kubernetes.io/projected/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-kube-api-access-2fzvg\") pod \"maas-api-revoke-sub-b48b994d-qztv9\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:10.099445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.099394 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.255978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.250621 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:10.642322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.633437 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:10.828322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.774847 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00605c20-be93-44bc-8e74-c80d43371bf2-kube-api-access-29fj5\" (UniqueName: \"kubernetes.io/projected/00605c20-be93-44bc-8e74-c80d43371bf2-kube-api-access-29fj5\") pod \"00605c20-be93-44bc-8e74-c80d43371bf2\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " Nov 03 22:29:10.847317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.839415 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-xks-ca\") pod \"00605c20-be93-44bc-8e74-c80d43371bf2\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " Nov 03 22:29:10.872318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.851431 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-openshift-ca\") pod \"00605c20-be93-44bc-8e74-c80d43371bf2\" (UID: \"00605c20-be93-44bc-8e74-c80d43371bf2\") " Nov 03 22:29:10.953318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.852723 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-openshift-ca" pod "00605c20-be93-44bc-8e74-c80d43371bf2" (UID: "00605c20-be93-44bc-8e74-c80d43371bf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:10.953318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.853618 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-xks-ca" pod "00605c20-be93-44bc-8e74-c80d43371bf2" (UID: "00605c20-be93-44bc-8e74-c80d43371bf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:10.953318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.909459 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00605c20-be93-44bc-8e74-c80d43371bf2-kube-api-access-29fj5" pod "00605c20-be93-44bc-8e74-c80d43371bf2" (UID: "00605c20-be93-44bc-8e74-c80d43371bf2"). InnerVolumeSpecName "kube-api-access-29fj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:10.953318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.908461 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:10.953318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:10.947175 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:29:11.090449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:11.090390 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:29:11.406635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:11.136486 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 03 22:29:11.409263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:11.191137 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29fj5\" (UniqueName: \"kubernetes.io/projected/00605c20-be93-44bc-8e74-c80d43371bf2-kube-api-access-29fj5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:11.409473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:11.409456 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:11.409568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:11.409555 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00605c20-be93-44bc-8e74-c80d43371bf2-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:11.409984 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:11.409967 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:29:12.174318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.164169 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.226320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.174343 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.226793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.175175 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.252317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.175350 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 22:29:12.370438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.175387 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Nov 03 22:29:12.396243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.175449 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.396597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.175561 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 03 22:29:12.402627 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.200121 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.402941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.226681 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.403188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.336997 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" Nov 03 22:29:12.403866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.337225 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:29:12.404091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.346825 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 22:29:12.406689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.406659 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.407264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.407206 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.430779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.430688 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.474107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.474063 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:12.866358 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:29:12.841210 4903 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.018s" Nov 03 22:29:12.866628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.866595 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt" event={"ID":"00605c20-be93-44bc-8e74-c80d43371bf2","Type":"ContainerDied","Data":"7a58496ca9e26d8fc0cf6a605a7604996e51b44c5af4ada2c55a0bfb622c028c"} Nov 03 22:29:12.866805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.866788 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a58496ca9e26d8fc0cf6a605a7604996e51b44c5af4ada2c55a0bfb622c028c" Nov 03 22:29:12.866954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.866942 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:29:12.969390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.969354 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9"] Nov 03 22:29:12.973861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:12.829930 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.125437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.125388 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.207339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.193592 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 22:29:13.212516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.212477 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 03 22:29:13.213114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.213092 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:29:13.273613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.273509 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.274272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.274249 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:29:13.396595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.396551 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.397222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.397201 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:29:13.432317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.416279 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.433697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.433657 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.434735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.434676 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 22:29:13.435253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.435237 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:29:13.475133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.475094 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.480413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.480378 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.484431 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.484382 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 03 22:29:13.492040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.484611 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:13.507867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.507814 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:29:14.098464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.508200 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:29:14.098850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.098829 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:29:14.105573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.105531 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.105986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:13.543255 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.133106 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:29:14.133060 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00605c20_be93_44bc_8e74_c80d43371bf2.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:29:14.171027 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.170958 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:14.183593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.183561 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:29:14.183784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.183641 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:29:14.183784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.183661 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:29:14.183784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.183679 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" event={"ID":"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b","Type":"ContainerStarted","Data":"f06ddd32e270a94b7c979c6fd05293d7069c89e4fc25018b7d029891f73a5654"} Nov 03 22:29:14.183784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.183730 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" event={"ID":"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b","Type":"ContainerStarted","Data":"0e3c56633d82f2338c482218d7407fdcc862c5d8006bb6cbb9a5bca7473b11da"} Nov 03 22:29:14.183963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.183816 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:29:14.183963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.183873 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:29:14.183963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.183948 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:29:14.184090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.183994 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:29:14.184090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.184058 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:29:14.184179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.184119 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:29:14.191656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.191632 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:29:14.191942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.191931 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:29:14.192086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.192077 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:29:14.207821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.207782 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:29:14.209624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.209329 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:29:14.219762 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.219742 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:29:14.698327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:14.698248 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" podStartSLOduration=6.698231272 podStartE2EDuration="6.698231272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:14.692232873 +0000 UTC m=+981.096173849" watchObservedRunningTime="2026-11-03 22:29:14.698231272 +0000 UTC m=+981.102172272" Nov 03 22:29:15.196801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:15.196771 4903 generic.go:356] "Generic (PLEG): container finished" podID="7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b" containerID="f06ddd32e270a94b7c979c6fd05293d7069c89e4fc25018b7d029891f73a5654" exitCode=0 Nov 03 22:29:15.197125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:15.197087 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" event={"ID":"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b","Type":"ContainerDied","Data":"f06ddd32e270a94b7c979c6fd05293d7069c89e4fc25018b7d029891f73a5654"} Nov 03 22:29:15.748336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:15.746869 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:16.392174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.392143 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:16.429232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.429199 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-kube-api-access-2fzvg\" (UniqueName: \"kubernetes.io/projected/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-kube-api-access-2fzvg\") pod \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " Nov 03 22:29:16.429453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.429375 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-xks-ca\") pod \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " Nov 03 22:29:16.429526 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.429485 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-openshift-ca\") pod \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\" (UID: \"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b\") " Nov 03 22:29:16.429995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.429895 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-openshift-ca" pod "7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b" (UID: "7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:16.430970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.430939 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:16.432605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.432495 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-kube-api-access-2fzvg" pod "7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b" (UID: "7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b"). InnerVolumeSpecName "kube-api-access-2fzvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:16.433448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.433418 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-xks-ca" pod "7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b" (UID: "7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:16.532061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.531976 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fzvg\" (UniqueName: \"kubernetes.io/projected/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-kube-api-access-2fzvg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:16.532061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:16.532009 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:17.221092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:17.221053 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" event={"ID":"7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b","Type":"ContainerDied","Data":"0e3c56633d82f2338c482218d7407fdcc862c5d8006bb6cbb9a5bca7473b11da"} Nov 03 22:29:17.221614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:17.221118 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e3c56633d82f2338c482218d7407fdcc862c5d8006bb6cbb9a5bca7473b11da" Nov 03 22:29:17.221614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:17.221188 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9" Nov 03 22:29:18.394131 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:29:18.394097 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:29:19.135339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.135305 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8"] Nov 03 22:29:19.136084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.136068 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b" containerName="revoke-keys" Nov 03 22:29:19.136134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.136085 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b" containerName="revoke-keys" Nov 03 22:29:19.136134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.136122 4903 cpu_manager.go:414] "containerMap: removing container" podUID="00605c20-be93-44bc-8e74-c80d43371bf2" containerName="revoke-keys" Nov 03 22:29:19.136134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.136127 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00605c20-be93-44bc-8e74-c80d43371bf2" containerName="revoke-keys" Nov 03 22:29:19.136237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.136232 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b" containerName="revoke-keys" Nov 03 22:29:19.136270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.136247 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="00605c20-be93-44bc-8e74-c80d43371bf2" containerName="revoke-keys" Nov 03 22:29:19.139970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.139951 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.173439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.173399 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:29:19.173642 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.173509 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:19.176160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.176128 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:29:19.176356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.174590 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:29:19.181273 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.181238 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8"] Nov 03 22:29:19.277797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.277763 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz"] Nov 03 22:29:19.282656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.282632 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.286832 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.286805 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a652e171-jmst8\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.286964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.286868 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a652e171-jmst8\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.287012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.286995 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkpl\" (UniqueName: \"kubernetes.io/projected/d5a06f81-9ca7-4966-9b90-c940be6aa899-kube-api-access-4vkpl\") pod \"maas-api-revoke-sub-a652e171-jmst8\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.301695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.301665 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz"] Nov 03 22:29:19.388880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.388795 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a652e171-jmst8\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.388880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.388877 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkpl\" (UniqueName: \"kubernetes.io/projected/d5a06f81-9ca7-4966-9b90-c940be6aa899-kube-api-access-4vkpl\") pod \"maas-api-revoke-sub-a652e171-jmst8\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.389128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.388934 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753b7f06-qh2zz\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.389128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.388995 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxz2\" (UniqueName: \"kubernetes.io/projected/55127199-7e7c-4b3a-b500-c344cc2d61c7-kube-api-access-4rxz2\") pod \"maas-api-revoke-sub-753b7f06-qh2zz\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.389234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.389139 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753b7f06-qh2zz\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.389234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.389161 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a652e171-jmst8\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.389854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.389833 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a652e171-jmst8\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.391332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.391313 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a652e171-jmst8\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.410664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.410631 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkpl\" (UniqueName: \"kubernetes.io/projected/d5a06f81-9ca7-4966-9b90-c940be6aa899-kube-api-access-4vkpl\") pod \"maas-api-revoke-sub-a652e171-jmst8\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.460946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.460917 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:19.490361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.490285 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxz2\" (UniqueName: \"kubernetes.io/projected/55127199-7e7c-4b3a-b500-c344cc2d61c7-kube-api-access-4rxz2\") pod \"maas-api-revoke-sub-753b7f06-qh2zz\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.490535 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.490472 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753b7f06-qh2zz\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.490713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.490695 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753b7f06-qh2zz\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.491628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.491602 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753b7f06-qh2zz\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.494748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.493952 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753b7f06-qh2zz\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.515079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.515046 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxz2\" (UniqueName: \"kubernetes.io/projected/55127199-7e7c-4b3a-b500-c344cc2d61c7-kube-api-access-4rxz2\") pod \"maas-api-revoke-sub-753b7f06-qh2zz\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.599283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.599247 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:19.772240 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:29:19.772212 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a06f81_9ca7_4966_9b90_c940be6aa899.slice/crio-f93e4e9eab2a0b3682aacab6be56d58b695032fcfc2cea2cd41c3ea97a5225e6 WatchSource:0}: Error finding container f93e4e9eab2a0b3682aacab6be56d58b695032fcfc2cea2cd41c3ea97a5225e6: Status 404 returned error can't find the container with id f93e4e9eab2a0b3682aacab6be56d58b695032fcfc2cea2cd41c3ea97a5225e6 Nov 03 22:29:19.774066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.774040 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8"] Nov 03 22:29:19.909031 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:29:19.909000 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55127199_7e7c_4b3a_b500_c344cc2d61c7.slice/crio-45e663dc54643cb7604ec4fec143b0bebe6814809f988aa89c42ae7d1c85ebbf WatchSource:0}: Error finding container 45e663dc54643cb7604ec4fec143b0bebe6814809f988aa89c42ae7d1c85ebbf: Status 404 returned error can't find the container with id 45e663dc54643cb7604ec4fec143b0bebe6814809f988aa89c42ae7d1c85ebbf Nov 03 22:29:19.910490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:19.910465 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz"] Nov 03 22:29:20.253062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:20.252985 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" event={"ID":"55127199-7e7c-4b3a-b500-c344cc2d61c7","Type":"ContainerStarted","Data":"45e663dc54643cb7604ec4fec143b0bebe6814809f988aa89c42ae7d1c85ebbf"} Nov 03 22:29:20.255107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:20.255075 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" event={"ID":"d5a06f81-9ca7-4966-9b90-c940be6aa899","Type":"ContainerStarted","Data":"431fa2f6a8f2919ac8cd58da46c59fd532260e2a6ca9276e6053b7517a77c5a2"} Nov 03 22:29:20.255274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:20.255129 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" event={"ID":"d5a06f81-9ca7-4966-9b90-c940be6aa899","Type":"ContainerStarted","Data":"f93e4e9eab2a0b3682aacab6be56d58b695032fcfc2cea2cd41c3ea97a5225e6"} Nov 03 22:29:20.298544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:20.298491 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" podStartSLOduration=1.298475517 podStartE2EDuration="1.298475517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:20.291173565 +0000 UTC m=+986.695114540" watchObservedRunningTime="2026-11-03 22:29:20.298475517 +0000 UTC m=+986.702416491" Nov 03 22:29:21.266039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:21.266009 4903 generic.go:356] "Generic (PLEG): container finished" podID="d5a06f81-9ca7-4966-9b90-c940be6aa899" containerID="431fa2f6a8f2919ac8cd58da46c59fd532260e2a6ca9276e6053b7517a77c5a2" exitCode=0 Nov 03 22:29:21.266502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:21.266101 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" event={"ID":"d5a06f81-9ca7-4966-9b90-c940be6aa899","Type":"ContainerDied","Data":"431fa2f6a8f2919ac8cd58da46c59fd532260e2a6ca9276e6053b7517a77c5a2"} Nov 03 22:29:21.267750 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:21.267728 4903 generic.go:356] "Generic (PLEG): container finished" podID="55127199-7e7c-4b3a-b500-c344cc2d61c7" containerID="a215eb1a20bcb14a61e40b951e31ed7c91abae9a3557020adb0008ce3e9d99e4" exitCode=0 Nov 03 22:29:21.267857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:21.267757 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" event={"ID":"55127199-7e7c-4b3a-b500-c344cc2d61c7","Type":"ContainerDied","Data":"a215eb1a20bcb14a61e40b951e31ed7c91abae9a3557020adb0008ce3e9d99e4"} Nov 03 22:29:22.461958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.461926 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:22.510233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.510206 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:22.537401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.537273 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a06f81-9ca7-4966-9b90-c940be6aa899-kube-api-access-4vkpl\" (UniqueName: \"kubernetes.io/projected/d5a06f81-9ca7-4966-9b90-c940be6aa899-kube-api-access-4vkpl\") pod \"d5a06f81-9ca7-4966-9b90-c940be6aa899\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " Nov 03 22:29:22.537785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.537762 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-openshift-ca\") pod \"d5a06f81-9ca7-4966-9b90-c940be6aa899\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " Nov 03 22:29:22.537991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.537973 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-xks-ca\") pod \"d5a06f81-9ca7-4966-9b90-c940be6aa899\" (UID: \"d5a06f81-9ca7-4966-9b90-c940be6aa899\") " Nov 03 22:29:22.538309 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.538246 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-openshift-ca" pod "d5a06f81-9ca7-4966-9b90-c940be6aa899" (UID: "d5a06f81-9ca7-4966-9b90-c940be6aa899"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:22.539282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.539253 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:22.560515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.554812 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a06f81-9ca7-4966-9b90-c940be6aa899-kube-api-access-4vkpl" pod "d5a06f81-9ca7-4966-9b90-c940be6aa899" (UID: "d5a06f81-9ca7-4966-9b90-c940be6aa899"). InnerVolumeSpecName "kube-api-access-4vkpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:22.560910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.560730 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-xks-ca" pod "d5a06f81-9ca7-4966-9b90-c940be6aa899" (UID: "d5a06f81-9ca7-4966-9b90-c940be6aa899"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:22.640104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.640068 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-xks-ca\") pod \"55127199-7e7c-4b3a-b500-c344cc2d61c7\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " Nov 03 22:29:22.640271 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.640232 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55127199-7e7c-4b3a-b500-c344cc2d61c7-kube-api-access-4rxz2\" (UniqueName: \"kubernetes.io/projected/55127199-7e7c-4b3a-b500-c344cc2d61c7-kube-api-access-4rxz2\") pod \"55127199-7e7c-4b3a-b500-c344cc2d61c7\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " Nov 03 22:29:22.640438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.640421 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-openshift-ca\") pod \"55127199-7e7c-4b3a-b500-c344cc2d61c7\" (UID: \"55127199-7e7c-4b3a-b500-c344cc2d61c7\") " Nov 03 22:29:22.640793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.640766 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-openshift-ca" pod "55127199-7e7c-4b3a-b500-c344cc2d61c7" (UID: "55127199-7e7c-4b3a-b500-c344cc2d61c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:22.640977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.640960 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vkpl\" (UniqueName: \"kubernetes.io/projected/d5a06f81-9ca7-4966-9b90-c940be6aa899-kube-api-access-4vkpl\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:22.641050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.640978 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:22.641050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.640987 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a06f81-9ca7-4966-9b90-c940be6aa899-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:22.642411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.642387 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55127199-7e7c-4b3a-b500-c344cc2d61c7-kube-api-access-4rxz2" pod "55127199-7e7c-4b3a-b500-c344cc2d61c7" (UID: "55127199-7e7c-4b3a-b500-c344cc2d61c7"). InnerVolumeSpecName "kube-api-access-4rxz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:22.642411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.642395 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-xks-ca" pod "55127199-7e7c-4b3a-b500-c344cc2d61c7" (UID: "55127199-7e7c-4b3a-b500-c344cc2d61c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:22.742313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.742274 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55127199-7e7c-4b3a-b500-c344cc2d61c7-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:22.742506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:22.742336 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rxz2\" (UniqueName: \"kubernetes.io/projected/55127199-7e7c-4b3a-b500-c344cc2d61c7-kube-api-access-4rxz2\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:23.287061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.287027 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" Nov 03 22:29:23.287263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.287066 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8" event={"ID":"d5a06f81-9ca7-4966-9b90-c940be6aa899","Type":"ContainerDied","Data":"f93e4e9eab2a0b3682aacab6be56d58b695032fcfc2cea2cd41c3ea97a5225e6"} Nov 03 22:29:23.287263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.287111 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93e4e9eab2a0b3682aacab6be56d58b695032fcfc2cea2cd41c3ea97a5225e6" Nov 03 22:29:23.288641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.288623 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" Nov 03 22:29:23.288735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.288647 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz" event={"ID":"55127199-7e7c-4b3a-b500-c344cc2d61c7","Type":"ContainerDied","Data":"45e663dc54643cb7604ec4fec143b0bebe6814809f988aa89c42ae7d1c85ebbf"} Nov 03 22:29:23.288735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.288683 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45e663dc54643cb7604ec4fec143b0bebe6814809f988aa89c42ae7d1c85ebbf" Nov 03 22:29:23.747206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.747176 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2"] Nov 03 22:29:23.748091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.748072 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d5a06f81-9ca7-4966-9b90-c940be6aa899" containerName="revoke-keys" Nov 03 22:29:23.748153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.748093 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a06f81-9ca7-4966-9b90-c940be6aa899" containerName="revoke-keys" Nov 03 22:29:23.748209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.748157 4903 cpu_manager.go:414] "containerMap: removing container" podUID="55127199-7e7c-4b3a-b500-c344cc2d61c7" containerName="revoke-keys" Nov 03 22:29:23.748209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.748165 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55127199-7e7c-4b3a-b500-c344cc2d61c7" containerName="revoke-keys" Nov 03 22:29:23.748346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.748331 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="55127199-7e7c-4b3a-b500-c344cc2d61c7" containerName="revoke-keys" Nov 03 22:29:23.748407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.748349 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a06f81-9ca7-4966-9b90-c940be6aa899" containerName="revoke-keys" Nov 03 22:29:23.752721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.752697 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.766957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.766927 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2"] Nov 03 22:29:23.769959 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.769937 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 22:29:23.770144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.770128 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 22:29:23.786337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.778702 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 22:29:23.786337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.778766 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h854s\"" Nov 03 22:29:23.856544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.856512 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.856731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.856635 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28264c75-3888-44b5-ad31-e2c60d8a2dbe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.856816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.856732 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.856816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.856800 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.856930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.856855 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.857021 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.857001 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.857181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.857164 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgnd\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-kube-api-access-ftgnd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.958859 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.958823 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28264c75-3888-44b5-ad31-e2c60d8a2dbe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.959036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.958880 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.959036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.958919 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.959036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.958962 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.959221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.959044 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.959221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.959142 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgnd\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-kube-api-access-ftgnd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.959350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.959249 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.960013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.959985 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.961634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.961610 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.961738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.961716 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28264c75-3888-44b5-ad31-e2c60d8a2dbe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.961833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.961811 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.961870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.961859 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:23.998004 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:23.997910 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgnd\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-kube-api-access-ftgnd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:24.001335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:24.001266 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-56cz2\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:24.069588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:24.069556 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:24.397981 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:29:24.397945 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28264c75_3888_44b5_ad31_e2c60d8a2dbe.slice/crio-5e7e133a95506d7f334270c956598f799f3949cce882fca68f3476060a3dbca6 WatchSource:0}: Error finding container 5e7e133a95506d7f334270c956598f799f3949cce882fca68f3476060a3dbca6: Status 404 returned error can't find the container with id 5e7e133a95506d7f334270c956598f799f3949cce882fca68f3476060a3dbca6 Nov 03 22:29:24.400682 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:24.400648 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:24.400793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:24.400724 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:24.401256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:24.401234 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2"] Nov 03 22:29:25.316318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:25.315356 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" event={"ID":"28264c75-3888-44b5-ad31-e2c60d8a2dbe","Type":"ContainerStarted","Data":"7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc"} Nov 03 22:29:25.316318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:25.316033 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" event={"ID":"28264c75-3888-44b5-ad31-e2c60d8a2dbe","Type":"ContainerStarted","Data":"5e7e133a95506d7f334270c956598f799f3949cce882fca68f3476060a3dbca6"} Nov 03 22:29:25.316318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:25.316075 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:25.322336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:25.317440 4903 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-56cz2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:29:25.322336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:25.317496 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" podUID="28264c75-3888-44b5-ad31-e2c60d8a2dbe" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:29:25.371539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:25.371487 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" podStartSLOduration=2.371472917 podStartE2EDuration="2.371472917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:25.36747093 +0000 UTC m=+991.771411906" watchObservedRunningTime="2026-11-03 22:29:25.371472917 +0000 UTC m=+991.775413892" Nov 03 22:29:26.326114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:26.326087 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 22:29:28.765617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.765582 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2"] Nov 03 22:29:28.770696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.770672 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.798331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.798287 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2"] Nov 03 22:29:28.942429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.942388 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.942777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.942752 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.942900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.942826 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.942900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.942886 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.943060 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.943033 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.943262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.943237 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.943415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.943327 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdmmn\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-kube-api-access-xdmmn\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.943710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.943684 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.943821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.943751 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:28.960262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.960227 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4"] Nov 03 22:29:28.973548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:28.973518 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.017288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.017207 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4"] Nov 03 22:29:29.041435 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.041402 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj"] Nov 03 22:29:29.047844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.047810 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.048070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.048040 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.048137 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.048123 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdmmn\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-kube-api-access-xdmmn\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.048328 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.048370 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.048406 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.048509 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.048567 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.048591 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.048641 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.049078 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-credential-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.049348 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-socket\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.049821 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-certs\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.050176 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-data\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.051339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.051129 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istiod-ca-cert\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.052052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.051592 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj"] Nov 03 22:29:29.057000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.056967 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-envoy\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.057143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.057056 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-podinfo\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.083871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.083577 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdmmn\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-kube-api-access-xdmmn\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.085114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.085087 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-token\") pod \"e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.091845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.091822 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:29.150393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150359 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.150516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150408 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.150516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150434 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.150516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150495 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57055112-7228-4604-929c-ef621a1a85af-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.150516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150515 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.150656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150544 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvmr\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-kube-api-access-sgvmr\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.150656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150605 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.150656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150625 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.150656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150648 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57055112-7228-4604-929c-ef621a1a85af-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.150786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150683 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a858a6a-0039-46e3-b77f-531dcaefb894-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.150786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150702 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.150786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150723 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.150786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150764 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.150903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150788 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.150903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150818 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gghp\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-kube-api-access-6gghp\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.150903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150847 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.150903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150869 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.150903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.150900 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.252124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252092 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvmr\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-kube-api-access-sgvmr\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.252284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252176 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.252284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252195 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.252284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252218 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57055112-7228-4604-929c-ef621a1a85af-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.252284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252255 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a858a6a-0039-46e3-b77f-531dcaefb894-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.252284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252272 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.252284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252305 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.252621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252354 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.252621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252378 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.252621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252413 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gghp\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-kube-api-access-6gghp\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.252621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252444 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.252621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252466 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.252621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252502 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.252621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252548 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.252621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252598 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.252983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252625 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.252983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252710 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57055112-7228-4604-929c-ef621a1a85af-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.252983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.252740 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.253138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.253115 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.253702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.253678 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.253953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.253928 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.255082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.254179 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.255082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.254415 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.255082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.254654 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a858a6a-0039-46e3-b77f-531dcaefb894-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.255082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.254705 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.255082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.254723 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.255082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.255030 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.255512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.255286 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57055112-7228-4604-929c-ef621a1a85af-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.257869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.257817 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.257869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.257831 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.260367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.260012 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57055112-7228-4604-929c-ef621a1a85af-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.261199 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.261170 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.275114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.275057 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvmr\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-kube-api-access-sgvmr\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.275447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.275428 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.275650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.275626 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.276801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.276782 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gghp\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-kube-api-access-6gghp\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.317140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.317111 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:29.433437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.433232 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2"] Nov 03 22:29:29.437093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.437058 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:29.437210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.437129 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:29.437210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.437186 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:29.448226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.448204 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:29.478280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.478254 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2"] Nov 03 22:29:29.484421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.484404 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.499730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.499663 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:29:29.499730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.499663 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:29:29.499730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.499710 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:29:29.503993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.503965 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2"] Nov 03 22:29:29.504456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.504427 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:29:29.561170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.560499 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhnxh\" (UniqueName: \"kubernetes.io/projected/4cb38697-525b-436f-bca0-ea1e391b112d-kube-api-access-bhnxh\") pod \"maas-api-revoke-sub-c9515d5c-fmvf2\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.561170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.560603 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9515d5c-fmvf2\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.561170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.560694 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9515d5c-fmvf2\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.648942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.648917 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4"] Nov 03 22:29:29.651344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.651258 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:29.651462 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.651389 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:29.651537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.651462 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:29.662041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.662014 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9515d5c-fmvf2\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.664029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.662154 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhnxh\" (UniqueName: \"kubernetes.io/projected/4cb38697-525b-436f-bca0-ea1e391b112d-kube-api-access-bhnxh\") pod \"maas-api-revoke-sub-c9515d5c-fmvf2\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.664029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.662353 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9515d5c-fmvf2\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.664029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.662739 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9515d5c-fmvf2\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.664874 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.664854 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9515d5c-fmvf2\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.684472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.684445 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhnxh\" (UniqueName: \"kubernetes.io/projected/4cb38697-525b-436f-bca0-ea1e391b112d-kube-api-access-bhnxh\") pod \"maas-api-revoke-sub-c9515d5c-fmvf2\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.800719 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.800637 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj"] Nov 03 22:29:29.806356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.806321 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:29.813646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.813605 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:29.813741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.813701 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:29.813832 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:29.813750 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:30.218637 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:29:30.218597 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb38697_525b_436f_bca0_ea1e391b112d.slice/crio-90a4655dfdf41000536ba8111b7bf7f591c14f09a8af8f57f8d2b92c0d91e62d WatchSource:0}: Error finding container 90a4655dfdf41000536ba8111b7bf7f591c14f09a8af8f57f8d2b92c0d91e62d: Status 404 returned error can't find the container with id 90a4655dfdf41000536ba8111b7bf7f591c14f09a8af8f57f8d2b92c0d91e62d Nov 03 22:29:30.231362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.225454 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2"] Nov 03 22:29:30.389276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.389165 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" event={"ID":"8ca6fd32-0245-4ea1-ae2c-81512aa4e979","Type":"ContainerStarted","Data":"a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb"} Nov 03 22:29:30.389276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.389232 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" event={"ID":"8ca6fd32-0245-4ea1-ae2c-81512aa4e979","Type":"ContainerStarted","Data":"e74902fd1e30571a172b1a8256ceca4d4e591335ac199e89ae5c6e337c42013d"} Nov 03 22:29:30.392888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.392858 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" event={"ID":"7a858a6a-0039-46e3-b77f-531dcaefb894","Type":"ContainerStarted","Data":"be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b"} Nov 03 22:29:30.393010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.392910 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" event={"ID":"7a858a6a-0039-46e3-b77f-531dcaefb894","Type":"ContainerStarted","Data":"3e5221161ac929d7e1cf9aeec6c5bcd72bbfec9db162de0b3a3c7813b25513e2"} Nov 03 22:29:30.394682 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.394655 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" event={"ID":"4cb38697-525b-436f-bca0-ea1e391b112d","Type":"ContainerStarted","Data":"90a4655dfdf41000536ba8111b7bf7f591c14f09a8af8f57f8d2b92c0d91e62d"} Nov 03 22:29:30.398320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.398262 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" event={"ID":"57055112-7228-4604-929c-ef621a1a85af","Type":"ContainerStarted","Data":"ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746"} Nov 03 22:29:30.398557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.398514 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" event={"ID":"57055112-7228-4604-929c-ef621a1a85af","Type":"ContainerStarted","Data":"15cb2f7ca80fa6eed7f6d5aebb5e1ec70e7842c32e08650ff9ad2257a95d3755"} Nov 03 22:29:30.436843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.436788 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" podStartSLOduration=2.43677288 podStartE2EDuration="2.43677288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:30.430842582 +0000 UTC m=+996.834783556" watchObservedRunningTime="2026-11-03 22:29:30.43677288 +0000 UTC m=+996.840713855" Nov 03 22:29:30.455448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.455355 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:30.455642 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.455624 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:29:30.455710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.455672 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:29:30.483370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.483252 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" podStartSLOduration=2.483236282 podStartE2EDuration="2.483236282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:30.479322334 +0000 UTC m=+996.883263307" watchObservedRunningTime="2026-11-03 22:29:30.483236282 +0000 UTC m=+996.887177257" Nov 03 22:29:30.536843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:30.536761 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podStartSLOduration=2.536736479 podStartE2EDuration="2.536736479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:30.527880242 +0000 UTC m=+996.931821222" watchObservedRunningTime="2026-11-03 22:29:30.536736479 +0000 UTC m=+996.940677456" Nov 03 22:29:31.095382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:31.093464 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:31.110914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:31.110869 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 22:29:31.111160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:31.111120 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 22:29:31.322513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:31.322464 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:31.327606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:31.327569 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:29:31.327772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:31.327636 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 22:29:31.430952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:31.430902 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" event={"ID":"4cb38697-525b-436f-bca0-ea1e391b112d","Type":"ContainerStarted","Data":"1031fd06c665c5b664f0e36778f147936aa3ce9c198058ce8ec5c4e05bd0032f"} Nov 03 22:29:31.467108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:31.466001 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:29:31.467108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:31.466065 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:29:32.096643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:32.096607 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 22:29:32.097255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:32.097223 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 22:29:32.319827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:32.319790 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:29:32.320092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:32.320067 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 22:29:32.450563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:32.450526 4903 generic.go:356] "Generic (PLEG): container finished" podID="4cb38697-525b-436f-bca0-ea1e391b112d" containerID="1031fd06c665c5b664f0e36778f147936aa3ce9c198058ce8ec5c4e05bd0032f" exitCode=0 Nov 03 22:29:32.454333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:32.453358 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" event={"ID":"4cb38697-525b-436f-bca0-ea1e391b112d","Type":"ContainerDied","Data":"1031fd06c665c5b664f0e36778f147936aa3ce9c198058ce8ec5c4e05bd0032f"} Nov 03 22:29:32.454895 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:32.454865 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:29:32.455000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:32.454919 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:29:33.094773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:33.094735 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 22:29:33.095058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:33.095030 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 22:29:33.324780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:33.324746 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:29:33.325370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:33.325340 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 22:29:33.458321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:33.458228 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:29:33.458321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:33.458315 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:29:34.129319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.117477 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 22:29:34.129319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.117564 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 22:29:34.345371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.341955 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:29:34.345371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.342017 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 22:29:34.488634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.488503 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7"] Nov 03 22:29:34.569462 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.569425 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:29:34.602504 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.602411 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:29:34.640018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.639985 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:34.929378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.901425 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7"] Nov 03 22:29:34.933110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.933070 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:34.937314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.933378 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:34.937580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:34.937555 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrdsx\" (UniqueName: \"kubernetes.io/projected/dd68d9d8-3853-4067-8d10-2c21049f28bc-kube-api-access-mrdsx\") pod \"maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:35.070964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.070923 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrdsx\" (UniqueName: \"kubernetes.io/projected/dd68d9d8-3853-4067-8d10-2c21049f28bc-kube-api-access-mrdsx\") pod \"maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:35.122523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.071746 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:35.122523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.113149 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:35.122523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.112696 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:35.122523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.121642 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 22:29:35.122523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.121883 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 22:29:35.138233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.138207 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:35.166365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.140836 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrdsx\" (UniqueName: \"kubernetes.io/projected/dd68d9d8-3853-4067-8d10-2c21049f28bc-kube-api-access-mrdsx\") pod \"maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:35.166365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.152668 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:35.166365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.153394 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" event={"ID":"4cb38697-525b-436f-bca0-ea1e391b112d","Type":"ContainerDied","Data":"90a4655dfdf41000536ba8111b7bf7f591c14f09a8af8f57f8d2b92c0d91e62d"} Nov 03 22:29:35.166365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.153448 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a4655dfdf41000536ba8111b7bf7f591c14f09a8af8f57f8d2b92c0d91e62d" Nov 03 22:29:35.198367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.195540 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:35.235226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.235198 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb"] Nov 03 22:29:35.329333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.329148 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef49c761-vwnnb"] Nov 03 22:29:35.330690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.330627 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:29:35.340414 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.340382 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 22:29:35.391371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.373611 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb38697-525b-436f-bca0-ea1e391b112d-kube-api-access-bhnxh\" (UniqueName: \"kubernetes.io/projected/4cb38697-525b-436f-bca0-ea1e391b112d-kube-api-access-bhnxh\") pod \"4cb38697-525b-436f-bca0-ea1e391b112d\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " Nov 03 22:29:35.391371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.373950 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-openshift-ca\") pod \"4cb38697-525b-436f-bca0-ea1e391b112d\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " Nov 03 22:29:35.391371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.385552 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-openshift-ca" pod "4cb38697-525b-436f-bca0-ea1e391b112d" (UID: "4cb38697-525b-436f-bca0-ea1e391b112d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:35.404328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.403088 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-xks-ca\") pod \"4cb38697-525b-436f-bca0-ea1e391b112d\" (UID: \"4cb38697-525b-436f-bca0-ea1e391b112d\") " Nov 03 22:29:35.425364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.419767 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:35.471510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.460496 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:29:35.471510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.460562 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:29:35.539753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.498837 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb38697-525b-436f-bca0-ea1e391b112d-kube-api-access-bhnxh" pod "4cb38697-525b-436f-bca0-ea1e391b112d" (UID: "4cb38697-525b-436f-bca0-ea1e391b112d"). InnerVolumeSpecName "kube-api-access-bhnxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:35.539753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.505814 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-xks-ca" pod "4cb38697-525b-436f-bca0-ea1e391b112d" (UID: "4cb38697-525b-436f-bca0-ea1e391b112d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:35.539753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.529899 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb38697-525b-436f-bca0-ea1e391b112d-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:35.539753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.529934 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhnxh\" (UniqueName: \"kubernetes.io/projected/4cb38697-525b-436f-bca0-ea1e391b112d-kube-api-access-bhnxh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:35.903969 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:35.903928 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca28ac88-227f-4a7c-84bb-78701735a634" path="/var/lib/kubelet/pods/ca28ac88-227f-4a7c-84bb-78701735a634/volumes" Nov 03 22:29:36.098687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:36.098648 4903 patch_prober.go:29] interesting pod/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Nov 03 22:29:36.103438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:36.103382 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Nov 03 22:29:36.216459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:36.216426 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2" Nov 03 22:29:36.308659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:36.307718 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7"] Nov 03 22:29:36.317756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:36.317724 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Nov 03 22:29:36.317880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:36.317784 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Nov 03 22:29:36.449099 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:36.449063 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:29:36.449516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:36.449127 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:29:37.263316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:37.262468 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" event={"ID":"dd68d9d8-3853-4067-8d10-2c21049f28bc","Type":"ContainerStarted","Data":"595c249376ec2da3fa0d9fd3738fc0c95303f3825f4945b1b9db4d86d4a6ad04"} Nov 03 22:29:37.263316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:37.262540 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" event={"ID":"dd68d9d8-3853-4067-8d10-2c21049f28bc","Type":"ContainerStarted","Data":"6b0b4b122b0e63a82ab3818924db70fb9c37a5e7418f17dcdb89a900fd6e14f3"} Nov 03 22:29:37.331728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:37.330894 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" podStartSLOduration=3.3308655959999998 podStartE2EDuration="3.330865596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:37.327858742 +0000 UTC m=+1003.731799720" watchObservedRunningTime="2026-11-03 22:29:37.330865596 +0000 UTC m=+1003.734806572" Nov 03 22:29:37.455144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:37.454750 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Nov 03 22:29:37.455144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:37.454817 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Nov 03 22:29:37.598264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:37.598230 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:37.600111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:37.600088 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:37.603697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:37.603646 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:29:38.009026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.008932 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:38.009631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.009609 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:38.025315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.021576 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:29:38.275576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.275495 4903 generic.go:356] "Generic (PLEG): container finished" podID="dd68d9d8-3853-4067-8d10-2c21049f28bc" containerID="595c249376ec2da3fa0d9fd3738fc0c95303f3825f4945b1b9db4d86d4a6ad04" exitCode=0 Nov 03 22:29:38.275732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.275574 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" event={"ID":"dd68d9d8-3853-4067-8d10-2c21049f28bc","Type":"ContainerDied","Data":"595c249376ec2da3fa0d9fd3738fc0c95303f3825f4945b1b9db4d86d4a6ad04"} Nov 03 22:29:38.453723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.453690 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:38.454237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.454218 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:38.455047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.455028 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:29:38.964938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.964899 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d"] Nov 03 22:29:38.965992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.965965 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4cb38697-525b-436f-bca0-ea1e391b112d" containerName="revoke-keys" Nov 03 22:29:38.965992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.965989 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb38697-525b-436f-bca0-ea1e391b112d" containerName="revoke-keys" Nov 03 22:29:38.966243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.966220 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb38697-525b-436f-bca0-ea1e391b112d" containerName="revoke-keys" Nov 03 22:29:38.970773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.970753 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:38.989844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:38.989807 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d"] Nov 03 22:29:39.086363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.086325 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwwn\" (UniqueName: \"kubernetes.io/projected/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-kube-api-access-6xwwn\") pod \"maas-api-revoke-keys-e2e-ait-1b279244-t8w8d\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.086574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.086446 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b279244-t8w8d\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.086574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.086499 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b279244-t8w8d\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.190361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.190322 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwwn\" (UniqueName: \"kubernetes.io/projected/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-kube-api-access-6xwwn\") pod \"maas-api-revoke-keys-e2e-ait-1b279244-t8w8d\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.190569 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.190417 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b279244-t8w8d\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.190569 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.190483 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b279244-t8w8d\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.191482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.191457 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b279244-t8w8d\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.193544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.193522 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b279244-t8w8d\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.244261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.244181 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwwn\" (UniqueName: \"kubernetes.io/projected/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-kube-api-access-6xwwn\") pod \"maas-api-revoke-keys-e2e-ait-1b279244-t8w8d\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.293549 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.293521 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:39.476894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.476866 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:39.598006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.597919 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd68d9d8-3853-4067-8d10-2c21049f28bc-kube-api-access-mrdsx\" (UniqueName: \"kubernetes.io/projected/dd68d9d8-3853-4067-8d10-2c21049f28bc-kube-api-access-mrdsx\") pod \"dd68d9d8-3853-4067-8d10-2c21049f28bc\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " Nov 03 22:29:39.598222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.598204 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-openshift-ca\") pod \"dd68d9d8-3853-4067-8d10-2c21049f28bc\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " Nov 03 22:29:39.598318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.598239 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-xks-ca\") pod \"dd68d9d8-3853-4067-8d10-2c21049f28bc\" (UID: \"dd68d9d8-3853-4067-8d10-2c21049f28bc\") " Nov 03 22:29:39.598697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.598666 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-openshift-ca" pod "dd68d9d8-3853-4067-8d10-2c21049f28bc" (UID: "dd68d9d8-3853-4067-8d10-2c21049f28bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:39.599089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.599063 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:39.602713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.602684 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd68d9d8-3853-4067-8d10-2c21049f28bc-kube-api-access-mrdsx" pod "dd68d9d8-3853-4067-8d10-2c21049f28bc" (UID: "dd68d9d8-3853-4067-8d10-2c21049f28bc"). InnerVolumeSpecName "kube-api-access-mrdsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:39.603994 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.603965 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-xks-ca" pod "dd68d9d8-3853-4067-8d10-2c21049f28bc" (UID: "dd68d9d8-3853-4067-8d10-2c21049f28bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:39.625280 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:29:39.625239 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3213a278_7de1_4ef1_bf1d_2dab3a7bfd59.slice/crio-72432909add0ea555dae6d5ce8b6246423ca4f24720d1c2ba5fbe1ca8b2c3e92 WatchSource:0}: Error finding container 72432909add0ea555dae6d5ce8b6246423ca4f24720d1c2ba5fbe1ca8b2c3e92: Status 404 returned error can't find the container with id 72432909add0ea555dae6d5ce8b6246423ca4f24720d1c2ba5fbe1ca8b2c3e92 Nov 03 22:29:39.629061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.628021 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d"] Nov 03 22:29:39.701169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.701137 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd68d9d8-3853-4067-8d10-2c21049f28bc-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:39.701169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:39.701166 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrdsx\" (UniqueName: \"kubernetes.io/projected/dd68d9d8-3853-4067-8d10-2c21049f28bc-kube-api-access-mrdsx\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:40.294454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:40.294414 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" event={"ID":"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59","Type":"ContainerStarted","Data":"7c3000b2aa3967c42459408497a82c6e07163dcbeb8fa440cf35625fe3cf137d"} Nov 03 22:29:40.294928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:40.294477 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" event={"ID":"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59","Type":"ContainerStarted","Data":"72432909add0ea555dae6d5ce8b6246423ca4f24720d1c2ba5fbe1ca8b2c3e92"} Nov 03 22:29:40.296187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:40.296151 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" Nov 03 22:29:40.296342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:40.296197 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7" event={"ID":"dd68d9d8-3853-4067-8d10-2c21049f28bc","Type":"ContainerDied","Data":"6b0b4b122b0e63a82ab3818924db70fb9c37a5e7418f17dcdb89a900fd6e14f3"} Nov 03 22:29:40.296342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:40.296237 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0b4b122b0e63a82ab3818924db70fb9c37a5e7418f17dcdb89a900fd6e14f3" Nov 03 22:29:40.341911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:40.341857 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" podStartSLOduration=2.341838784 podStartE2EDuration="2.341838784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:40.330876104 +0000 UTC m=+1006.734817078" watchObservedRunningTime="2026-11-03 22:29:40.341838784 +0000 UTC m=+1006.745779762" Nov 03 22:29:41.307976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:41.307937 4903 generic.go:356] "Generic (PLEG): container finished" podID="3213a278-7de1-4ef1-bf1d-2dab3a7bfd59" containerID="7c3000b2aa3967c42459408497a82c6e07163dcbeb8fa440cf35625fe3cf137d" exitCode=0 Nov 03 22:29:41.308419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:41.308028 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" event={"ID":"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59","Type":"ContainerDied","Data":"7c3000b2aa3967c42459408497a82c6e07163dcbeb8fa440cf35625fe3cf137d"} Nov 03 22:29:42.577849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.577779 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:42.649284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.649245 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-openshift-ca\") pod \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " Nov 03 22:29:42.649501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.649474 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-xks-ca\") pod \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " Nov 03 22:29:42.649581 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.649537 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-kube-api-access-6xwwn\" (UniqueName: \"kubernetes.io/projected/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-kube-api-access-6xwwn\") pod \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\" (UID: \"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59\") " Nov 03 22:29:42.650121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.650090 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-openshift-ca" pod "3213a278-7de1-4ef1-bf1d-2dab3a7bfd59" (UID: "3213a278-7de1-4ef1-bf1d-2dab3a7bfd59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:42.659237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.652361 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-xks-ca" pod "3213a278-7de1-4ef1-bf1d-2dab3a7bfd59" (UID: "3213a278-7de1-4ef1-bf1d-2dab3a7bfd59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:29:42.659237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.652719 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-kube-api-access-6xwwn" pod "3213a278-7de1-4ef1-bf1d-2dab3a7bfd59" (UID: "3213a278-7de1-4ef1-bf1d-2dab3a7bfd59"). InnerVolumeSpecName "kube-api-access-6xwwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:29:42.752504 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.752466 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:42.752504 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.752501 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xwwn\" (UniqueName: \"kubernetes.io/projected/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-kube-api-access-6xwwn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:42.752754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:42.752518 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:29:43.331480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:43.331448 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" Nov 03 22:29:43.331696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:43.331472 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d" event={"ID":"3213a278-7de1-4ef1-bf1d-2dab3a7bfd59","Type":"ContainerDied","Data":"72432909add0ea555dae6d5ce8b6246423ca4f24720d1c2ba5fbe1ca8b2c3e92"} Nov 03 22:29:43.331696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:43.331525 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72432909add0ea555dae6d5ce8b6246423ca4f24720d1c2ba5fbe1ca8b2c3e92" Nov 03 22:29:44.032589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.032552 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q"] Nov 03 22:29:44.033649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.033627 4903 cpu_manager.go:414] "containerMap: removing container" podUID="dd68d9d8-3853-4067-8d10-2c21049f28bc" containerName="revoke-keys" Nov 03 22:29:44.033649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.033650 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd68d9d8-3853-4067-8d10-2c21049f28bc" containerName="revoke-keys" Nov 03 22:29:44.033830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.033670 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3213a278-7de1-4ef1-bf1d-2dab3a7bfd59" containerName="revoke-keys" Nov 03 22:29:44.033830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.033678 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3213a278-7de1-4ef1-bf1d-2dab3a7bfd59" containerName="revoke-keys" Nov 03 22:29:44.033935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.033884 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd68d9d8-3853-4067-8d10-2c21049f28bc" containerName="revoke-keys" Nov 03 22:29:44.033935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.033896 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3213a278-7de1-4ef1-bf1d-2dab3a7bfd59" containerName="revoke-keys" Nov 03 22:29:44.040192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.040169 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.045485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.045456 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q"] Nov 03 22:29:44.178248 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.178212 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.178472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.178392 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.178472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.178455 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.178601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.178493 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.178601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.178527 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.178697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.178616 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.178747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.178730 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.178816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.178788 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.178888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.178849 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q77t\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-kube-api-access-6q77t\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.279804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.279766 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.279804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.279805 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q77t\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-kube-api-access-6q77t\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.279868 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.279982 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280033 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280068 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280095 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280185 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280327 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280419 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-credential-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280450 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-socket\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280530 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-data\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.280609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280567 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-certs\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.281006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.280985 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istiod-ca-cert\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.282356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.282334 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-envoy\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.282646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.282601 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-podinfo\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.301451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.301425 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-token\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.301586 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.301564 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q77t\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-kube-api-access-6q77t\") pod \"e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.359013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.358980 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:44.693554 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:29:44.693525 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6096f0_23ba_4115_9f27_84fbd9765f1b.slice/crio-ae2d2ac0a6aca9d98280d2d2491515e1c5bd6e4340f274499286e2a63359f843 WatchSource:0}: Error finding container ae2d2ac0a6aca9d98280d2d2491515e1c5bd6e4340f274499286e2a63359f843: Status 404 returned error can't find the container with id ae2d2ac0a6aca9d98280d2d2491515e1c5bd6e4340f274499286e2a63359f843 Nov 03 22:29:44.695918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.695890 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q"] Nov 03 22:29:44.696286 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.696252 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:44.696396 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.696348 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:44.696454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:44.696395 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:29:45.359921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:45.359879 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" event={"ID":"0e6096f0-23ba-4115-9f27-84fbd9765f1b","Type":"ContainerStarted","Data":"db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb"} Nov 03 22:29:45.360417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:45.359948 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" event={"ID":"0e6096f0-23ba-4115-9f27-84fbd9765f1b","Type":"ContainerStarted","Data":"ae2d2ac0a6aca9d98280d2d2491515e1c5bd6e4340f274499286e2a63359f843"} Nov 03 22:29:45.406646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:45.406592 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podStartSLOduration=2.406576714 podStartE2EDuration="2.406576714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:29:45.402505671 +0000 UTC m=+1011.806446656" watchObservedRunningTime="2026-11-03 22:29:45.406576714 +0000 UTC m=+1011.810517689" Nov 03 22:29:46.365964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:46.365855 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:46.368476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:46.368448 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:29:46.368628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:46.368505 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 22:29:47.366036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:47.365990 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:29:47.366584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:47.366061 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 22:29:48.381111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:48.381076 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:29:48.381751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:48.381718 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 22:29:48.430436 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:29:48.430402 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:29:48.623280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:48.623251 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:29:49.367313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:49.359679 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:29:49.367313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:49.359742 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 22:29:49.670708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:49.670102 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t"] Nov 03 22:29:50.371236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:50.370762 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:29:50.371236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:50.370829 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 22:29:50.927434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:50.927384 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:51.363757 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:51.363658 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7"] Nov 03 22:29:51.394223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:51.394184 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:29:51.394535 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:51.394503 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 22:29:51.407320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:51.404424 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b7b10505-s84m7"] Nov 03 22:29:51.741314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:51.740543 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd68d9d8-3853-4067-8d10-2c21049f28bc" path="/var/lib/kubelet/pods/dd68d9d8-3853-4067-8d10-2c21049f28bc/volumes" Nov 03 22:29:52.360157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:52.360116 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 22:29:52.360853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:52.360812 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 22:29:53.913478 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:53.913425 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:54.402447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:54.399119 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:54.470315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:54.460533 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:54.696701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:54.672440 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" podUID="788c17f5-3df0-45eb-b35c-390eddb3beeb" containerName="payload-pre-processing" containerID="cri-o://80de6e25c0c1d7edad5ebe796dce3f85d478699179fffe7427995dd38a403868" gracePeriod=25 Nov 03 22:29:54.969552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:54.931509 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 22:29:54.969552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:54.968025 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:54.978503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:54.978350 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:29:55.007720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.007680 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.040268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.040227 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.243755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.243640 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.270802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.243991 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.302090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.244035 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:29:55.302090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.271522 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.302090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.244083 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.302090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.295708 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 22:29:55.302090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.296924 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 03 22:29:55.302090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.244978 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Nov 03 22:29:55.302090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.298818 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 03 22:29:55.302090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.298875 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:29:55.324349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.312916 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.488319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.479815 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:55.488319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.479880 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:55.548765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.532807 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.025318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:55.954400 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 22:29:56.140141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.140095 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.173582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.173536 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 03 22:29:56.187518 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:29:56.187491 4903 logging.go:55] [core] [Channel #1808 SubChannel #1809]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.128:50052", ServerName: "10.132.0.128:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.128:50052: connect: connection refused" Nov 03 22:29:56.210350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.210284 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 22:29:56.214816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.214762 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.300373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.274605 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 22:29:56.300373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.275005 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.300373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.275277 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:29:56.300373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.280477 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:29:56.300373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.280847 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.300884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.300847 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.301404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.301356 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 22:29:56.328760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.305620 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 03 22:29:56.354319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.328318 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Nov 03 22:29:56.384317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.383647 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.525002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.524964 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:29:56.627400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.627320 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:29:56.654455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.654420 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:29:56.654736 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.654720 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:29:56.654880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.654869 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:29:56.657487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.657458 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" event={"ID":"788c17f5-3df0-45eb-b35c-390eddb3beeb","Type":"ContainerDied","Data":"80de6e25c0c1d7edad5ebe796dce3f85d478699179fffe7427995dd38a403868"} Nov 03 22:29:56.657734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.657711 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:29:56.683463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.683430 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:29:56.685386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.685361 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:29:56.686418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.686402 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:29:56.687216 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.687201 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:29:56.687962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.687946 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:29:56.697081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.697062 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:29:56.697680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.697664 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:29:56.705154 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.550391 4903 patch_prober.go:29] interesting pod/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:56.705366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.705342 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.705764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.536413 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:56.705952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.585516 4903 generic.go:356] "Generic (PLEG): container finished" podID="788c17f5-3df0-45eb-b35c-390eddb3beeb" containerID="80de6e25c0c1d7edad5ebe796dce3f85d478699179fffe7427995dd38a403868" exitCode=0 Nov 03 22:29:56.722466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.722441 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:29:56.807985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:56.807943 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" podUID="788c17f5-3df0-45eb-b35c-390eddb3beeb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.128:50052\" within 1s: context deadline exceeded" Nov 03 22:29:57.028068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.028027 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.270394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.267198 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 03 22:29:57.270394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.268108 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.313370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.273420 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 22:29:57.313370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.274462 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 22:29:57.413316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.373975 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:57.440328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.402385 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.464681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.404526 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.491355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.404554 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.491968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.404585 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 03 22:29:57.501355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.404606 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Nov 03 22:29:57.527792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.404625 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded" Nov 03 22:29:57.528034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.404952 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 22:29:57.591389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.404981 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:57.591986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.405002 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:29:57.592424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.405022 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:29:57.621848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.405041 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded" Nov 03 22:29:58.212327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.446457 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:58.374979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.338865 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:29:58.374979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.986365 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 03 22:29:58.430817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.430775 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 22:29:58.658318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.654077 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:58.658318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.656339 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:58.696316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.684526 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 22:29:58.708265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.708215 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Nov 03 22:29:58.712421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.712280 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:58.724416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.708611 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:58.725721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.709262 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 22:29:58.729327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.709843 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 22:29:58.736411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.709885 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 03 22:29:58.737207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.710248 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Nov 03 22:29:58.738345 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.710275 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 22:29:58.739590 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.711185 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 03 22:29:58.745790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.711225 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 22:29:58.941374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:57.886600 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:58.996317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:58.995429 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-7bb9489676-bnqlr" podUID="76f24aa5-5e88-488f-90e3-252e1e114b39" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 03 22:29:59.007279 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:29:59.007231 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788c17f5_3df0_45eb_b35c_390eddb3beeb.slice/crio-80de6e25c0c1d7edad5ebe796dce3f85d478699179fffe7427995dd38a403868.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788c17f5_3df0_45eb_b35c_390eddb3beeb.slice/crio-conmon-80de6e25c0c1d7edad5ebe796dce3f85d478699179fffe7427995dd38a403868.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:29:59.020102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.020048 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2z58q" podUID="a8b9b4d3-1dde-4142-84e9-3391d26a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.124646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.124608 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:29:59.125049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.124943 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.199599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.199505 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7" podUID="c9d72039-62c4-4d01-85f9-4e65d431a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.217436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.217383 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.386579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.386538 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:29:59.599420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.587217 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 22:29:59.654478 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.654416 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.695486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.695438 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.704868 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:29:59.704835 4903 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.366s" Nov 03 22:29:59.722321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.714726 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:29:59.722321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.715055 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 22:29:59.722321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.715594 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 22:29:59.732817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.732769 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 03 22:29:59.735852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.735802 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:29:59.736388 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.736359 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.749324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.742016 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 22:29:59.760321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.758697 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.760321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.758968 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Nov 03 22:29:59.762284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.762255 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788c17f5-3df0-45eb-b35c-390eddb3beeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788c17f5-3df0-45eb-b35c-390eddb3beeb-plugins-config-volume\") pod \"788c17f5-3df0-45eb-b35c-390eddb3beeb\" (UID: \"788c17f5-3df0-45eb-b35c-390eddb3beeb\") " Nov 03 22:29:59.762478 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.762463 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788c17f5-3df0-45eb-b35c-390eddb3beeb-kube-api-access-xtfbz\" (UniqueName: \"kubernetes.io/projected/788c17f5-3df0-45eb-b35c-390eddb3beeb-kube-api-access-xtfbz\") pod \"788c17f5-3df0-45eb-b35c-390eddb3beeb\" (UID: \"788c17f5-3df0-45eb-b35c-390eddb3beeb\") " Nov 03 22:29:59.858013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.832520 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.858013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.837832 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788c17f5-3df0-45eb-b35c-390eddb3beeb-plugins-config-volume" pod "788c17f5-3df0-45eb-b35c-390eddb3beeb" (UID: "788c17f5-3df0-45eb-b35c-390eddb3beeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:29:59.858013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.854790 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:29:59.981519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:29:59.854875 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788c17f5-3df0-45eb-b35c-390eddb3beeb-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:00.049355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.047058 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.109477 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.109552 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.109593 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.110650 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.113590 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.114685 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.126435 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.131003 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.131333 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 22:30:00.175192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.132261 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:30:00.192417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.192138 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" Nov 03 22:30:00.194624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.193569 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788c17f5-3df0-45eb-b35c-390eddb3beeb-kube-api-access-xtfbz" pod "788c17f5-3df0-45eb-b35c-390eddb3beeb" (UID: "788c17f5-3df0-45eb-b35c-390eddb3beeb"). InnerVolumeSpecName "kube-api-access-xtfbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:00.194624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.193966 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:30:00.213090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.213059 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtfbz\" (UniqueName: \"kubernetes.io/projected/788c17f5-3df0-45eb-b35c-390eddb3beeb-kube-api-access-xtfbz\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:00.232315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.231391 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" event={"ID":"788c17f5-3df0-45eb-b35c-390eddb3beeb","Type":"ContainerDied","Data":"06113159bab69c4d1a7b414d59c796180df3257087f2b5b05e304791474dd773"} Nov 03 22:30:00.232315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.231473 4903 scope.go:122] "RemoveContainer" containerID="80de6e25c0c1d7edad5ebe796dce3f85d478699179fffe7427995dd38a403868" Nov 03 22:30:00.232315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.231524 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t" Nov 03 22:30:00.388353 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.367643 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d"] Nov 03 22:30:00.390951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.390917 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b279244-t8w8d"] Nov 03 22:30:00.542745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.542704 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s"] Nov 03 22:30:00.544031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.543991 4903 cpu_manager.go:414] "containerMap: removing container" podUID="788c17f5-3df0-45eb-b35c-390eddb3beeb" containerName="payload-pre-processing" Nov 03 22:30:00.544200 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.544182 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788c17f5-3df0-45eb-b35c-390eddb3beeb" containerName="payload-pre-processing" Nov 03 22:30:00.544610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.544592 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="788c17f5-3df0-45eb-b35c-390eddb3beeb" containerName="payload-pre-processing" Nov 03 22:30:00.570579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.570545 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm"] Nov 03 22:30:00.571062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.570851 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:00.589624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.589596 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq"] Nov 03 22:30:00.590966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.590561 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:30:00.590966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.590832 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:30:00.591883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.591862 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:00.602204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.602181 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:00.608019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.607993 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d"] Nov 03 22:30:00.650512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.650479 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq"] Nov 03 22:30:00.650691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.650533 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s"] Nov 03 22:30:00.650691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.650632 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:00.661787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.661760 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm"] Nov 03 22:30:00.677336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.677287 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj"] Nov 03 22:30:00.677554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.677532 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:30:00.681703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.681672 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446e624-f0c0-4901-8542-a332384a8b82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq\" (UID: \"d446e624-f0c0-4901-8542-a332384a8b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:00.681823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.681792 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w442f\" (UniqueName: \"kubernetes.io/projected/21250882-c6f7-4fb4-ab40-70235609e3d4-kube-api-access-w442f\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm\" (UID: \"21250882-c6f7-4fb4-ab40-70235609e3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:00.681892 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.681875 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s\" (UID: \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:00.681956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.681941 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plj9v\" (UniqueName: \"kubernetes.io/projected/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-kube-api-access-plj9v\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s\" (UID: \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:00.682033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.682016 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21250882-c6f7-4fb4-ab40-70235609e3d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm\" (UID: \"21250882-c6f7-4fb4-ab40-70235609e3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:00.682156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.682140 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz8ls\" (UniqueName: \"kubernetes.io/projected/d446e624-f0c0-4901-8542-a332384a8b82-kube-api-access-nz8ls\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq\" (UID: \"d446e624-f0c0-4901-8542-a332384a8b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:00.695539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.695484 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d"] Nov 03 22:30:00.695693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.695611 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:00.760315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.733272 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj"] Nov 03 22:30:00.781313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.765790 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj"] Nov 03 22:30:00.781313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.765941 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:00.789908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.789871 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c89c6\" (UniqueName: \"kubernetes.io/projected/06af2cbf-605f-41db-83be-08545aaa5b31-kube-api-access-c89c6\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d\" (UID: \"06af2cbf-605f-41db-83be-08545aaa5b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:00.790079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.789946 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plj9v\" (UniqueName: \"kubernetes.io/projected/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-kube-api-access-plj9v\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s\" (UID: \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:00.790079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.790030 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21250882-c6f7-4fb4-ab40-70235609e3d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm\" (UID: \"21250882-c6f7-4fb4-ab40-70235609e3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:00.790201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.790105 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06af2cbf-605f-41db-83be-08545aaa5b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d\" (UID: \"06af2cbf-605f-41db-83be-08545aaa5b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:00.790201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.790191 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8ls\" (UniqueName: \"kubernetes.io/projected/d446e624-f0c0-4901-8542-a332384a8b82-kube-api-access-nz8ls\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq\" (UID: \"d446e624-f0c0-4901-8542-a332384a8b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:00.790320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.790235 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg5r5\" (UniqueName: \"kubernetes.io/projected/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-kube-api-access-cg5r5\") pod \"maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj\" (UID: \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:00.790380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.790367 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj\" (UID: \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:00.790466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.790444 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446e624-f0c0-4901-8542-a332384a8b82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq\" (UID: \"d446e624-f0c0-4901-8542-a332384a8b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:00.790613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.790596 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w442f\" (UniqueName: \"kubernetes.io/projected/21250882-c6f7-4fb4-ab40-70235609e3d4-kube-api-access-w442f\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm\" (UID: \"21250882-c6f7-4fb4-ab40-70235609e3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:00.790709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.790695 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s\" (UID: \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:00.799443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.799401 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21250882-c6f7-4fb4-ab40-70235609e3d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm\" (UID: \"21250882-c6f7-4fb4-ab40-70235609e3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:00.802739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.800505 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446e624-f0c0-4901-8542-a332384a8b82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq\" (UID: \"d446e624-f0c0-4901-8542-a332384a8b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:00.802739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.801443 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s\" (UID: \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:00.852747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.851120 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj"] Nov 03 22:30:00.895320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.893523 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj\" (UID: \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:00.895320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.893776 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b112bfd-7ef9-4e26-954c-3627c4a3e648-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-mzmlj\" (UID: \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:00.895320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.893888 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c89c6\" (UniqueName: \"kubernetes.io/projected/06af2cbf-605f-41db-83be-08545aaa5b31-kube-api-access-c89c6\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d\" (UID: \"06af2cbf-605f-41db-83be-08545aaa5b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:00.895320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.894056 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06af2cbf-605f-41db-83be-08545aaa5b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d\" (UID: \"06af2cbf-605f-41db-83be-08545aaa5b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:00.895320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.894131 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fpn\" (UniqueName: \"kubernetes.io/projected/5b112bfd-7ef9-4e26-954c-3627c4a3e648-kube-api-access-m5fpn\") pod \"maas-api-key-cleanup-29895750-mzmlj\" (UID: \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:00.895320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.894195 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5r5\" (UniqueName: \"kubernetes.io/projected/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-kube-api-access-cg5r5\") pod \"maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj\" (UID: \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:00.897971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.897550 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06af2cbf-605f-41db-83be-08545aaa5b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d\" (UID: \"06af2cbf-605f-41db-83be-08545aaa5b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:00.898996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.898857 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj\" (UID: \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:00.911710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.911675 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plj9v\" (UniqueName: \"kubernetes.io/projected/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-kube-api-access-plj9v\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s\" (UID: \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:00.913739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.913711 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8ls\" (UniqueName: \"kubernetes.io/projected/d446e624-f0c0-4901-8542-a332384a8b82-kube-api-access-nz8ls\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq\" (UID: \"d446e624-f0c0-4901-8542-a332384a8b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:00.918877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:00.918849 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w442f\" (UniqueName: \"kubernetes.io/projected/21250882-c6f7-4fb4-ab40-70235609e3d4-kube-api-access-w442f\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm\" (UID: \"21250882-c6f7-4fb4-ab40-70235609e3d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:01.029116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.029029 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5r5\" (UniqueName: \"kubernetes.io/projected/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-kube-api-access-cg5r5\") pod \"maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj\" (UID: \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:01.039326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.039267 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c89c6\" (UniqueName: \"kubernetes.io/projected/06af2cbf-605f-41db-83be-08545aaa5b31-kube-api-access-c89c6\") pod \"maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d\" (UID: \"06af2cbf-605f-41db-83be-08545aaa5b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:01.076150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.040683 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b112bfd-7ef9-4e26-954c-3627c4a3e648-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-mzmlj\" (UID: \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:01.076704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.076673 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fpn\" (UniqueName: \"kubernetes.io/projected/5b112bfd-7ef9-4e26-954c-3627c4a3e648-kube-api-access-m5fpn\") pod \"maas-api-key-cleanup-29895750-mzmlj\" (UID: \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:01.077424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.042923 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:01.077903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.045004 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:01.078049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.059041 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:01.078188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.041496 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b112bfd-7ef9-4e26-954c-3627c4a3e648-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895750-mzmlj\" (UID: \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:01.094453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.094350 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t"] Nov 03 22:30:01.099422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.099383 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fpn\" (UniqueName: \"kubernetes.io/projected/5b112bfd-7ef9-4e26-954c-3627c4a3e648-kube-api-access-m5fpn\") pod \"maas-api-key-cleanup-29895750-mzmlj\" (UID: \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:01.107546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.107513 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b279244-55d699df5f-8hq7t"] Nov 03 22:30:01.198686 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.198635 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:01.232327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.209257 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:01.300034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.295512 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:01.767695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.767651 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3213a278-7de1-4ef1-bf1d-2dab3a7bfd59" path="/var/lib/kubelet/pods/3213a278-7de1-4ef1-bf1d-2dab3a7bfd59/volumes" Nov 03 22:30:01.768287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.768264 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788c17f5-3df0-45eb-b35c-390eddb3beeb" path="/var/lib/kubelet/pods/788c17f5-3df0-45eb-b35c-390eddb3beeb/volumes" Nov 03 22:30:01.831878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.831845 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj"] Nov 03 22:30:01.833377 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:30:01.833345 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c583acd_e85b_4b5c_83d8_0b5e7774dbc9.slice/crio-99d8f86ea7f5e5c35fe2acdb5f57ac6e33929e409e352f8d859f81edbc8430dc WatchSource:0}: Error finding container 99d8f86ea7f5e5c35fe2acdb5f57ac6e33929e409e352f8d859f81edbc8430dc: Status 404 returned error can't find the container with id 99d8f86ea7f5e5c35fe2acdb5f57ac6e33929e409e352f8d859f81edbc8430dc Nov 03 22:30:01.848917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.848364 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm"] Nov 03 22:30:01.855188 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:30:01.855158 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21250882_c6f7_4fb4_ab40_70235609e3d4.slice/crio-2bcaf5630443103024c7b4fffc8d4dcf07266b01961dd5068298a98f4ed8057e WatchSource:0}: Error finding container 2bcaf5630443103024c7b4fffc8d4dcf07266b01961dd5068298a98f4ed8057e: Status 404 returned error can't find the container with id 2bcaf5630443103024c7b4fffc8d4dcf07266b01961dd5068298a98f4ed8057e Nov 03 22:30:01.901100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.901070 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d"] Nov 03 22:30:01.906934 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:30:01.906890 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06af2cbf_605f_41db_83be_08545aaa5b31.slice/crio-d06090fc2126c7df2a80912f3176b350edc8030d56ec849f4baaadc0708c9004 WatchSource:0}: Error finding container d06090fc2126c7df2a80912f3176b350edc8030d56ec849f4baaadc0708c9004: Status 404 returned error can't find the container with id d06090fc2126c7df2a80912f3176b350edc8030d56ec849f4baaadc0708c9004 Nov 03 22:30:01.910780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.910737 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq"] Nov 03 22:30:01.923619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.923583 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj"] Nov 03 22:30:01.929412 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:30:01.929381 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b112bfd_7ef9_4e26_954c_3627c4a3e648.slice/crio-8bc5336108b46388312b6faba6d4c2736ebfe8fb2d523e9e968dfd6090b53336 WatchSource:0}: Error finding container 8bc5336108b46388312b6faba6d4c2736ebfe8fb2d523e9e968dfd6090b53336: Status 404 returned error can't find the container with id 8bc5336108b46388312b6faba6d4c2736ebfe8fb2d523e9e968dfd6090b53336 Nov 03 22:30:01.966689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:01.966657 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s"] Nov 03 22:30:02.376899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:02.376858 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" event={"ID":"5b112bfd-7ef9-4e26-954c-3627c4a3e648","Type":"ContainerStarted","Data":"8bc5336108b46388312b6faba6d4c2736ebfe8fb2d523e9e968dfd6090b53336"} Nov 03 22:30:02.388388 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:02.388328 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" event={"ID":"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6","Type":"ContainerStarted","Data":"0f2d41aed26039b75ea2972877d0a96fc8e8a4d8f6a16348ed58112e71781f5f"} Nov 03 22:30:02.392838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:02.392549 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" event={"ID":"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9","Type":"ContainerStarted","Data":"99d8f86ea7f5e5c35fe2acdb5f57ac6e33929e409e352f8d859f81edbc8430dc"} Nov 03 22:30:02.396698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:02.396663 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" event={"ID":"d446e624-f0c0-4901-8542-a332384a8b82","Type":"ContainerStarted","Data":"10fa75358af59046e69b8eaf48594da559182f34ce9b1b37dd67ebf7ea8bcbdb"} Nov 03 22:30:02.398026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:02.397988 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" event={"ID":"06af2cbf-605f-41db-83be-08545aaa5b31","Type":"ContainerStarted","Data":"d06090fc2126c7df2a80912f3176b350edc8030d56ec849f4baaadc0708c9004"} Nov 03 22:30:02.400102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:02.400073 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" event={"ID":"21250882-c6f7-4fb4-ab40-70235609e3d4","Type":"ContainerStarted","Data":"2bcaf5630443103024c7b4fffc8d4dcf07266b01961dd5068298a98f4ed8057e"} Nov 03 22:30:03.432610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.432571 4903 generic.go:356] "Generic (PLEG): container finished" podID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerID="4b0c2a1298efd0517d542e15f5454309af8c0ad2e2268735c3bf73016ee27493" exitCode=60 Nov 03 22:30:03.433263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.433241 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" event={"ID":"5b112bfd-7ef9-4e26-954c-3627c4a3e648","Type":"ContainerDied","Data":"4b0c2a1298efd0517d542e15f5454309af8c0ad2e2268735c3bf73016ee27493"} Nov 03 22:30:03.433648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.433030 4903 scope.go:122] "RemoveContainer" containerID="4b0c2a1298efd0517d542e15f5454309af8c0ad2e2268735c3bf73016ee27493" Nov 03 22:30:03.442362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.440020 4903 generic.go:356] "Generic (PLEG): container finished" podID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerID="11d4f629db7cc26e0fb5b23aa47a74df47cc3dca4857b9ede2f9d8f9e2a5148e" exitCode=60 Nov 03 22:30:03.442362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.440176 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" event={"ID":"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6","Type":"ContainerDied","Data":"11d4f629db7cc26e0fb5b23aa47a74df47cc3dca4857b9ede2f9d8f9e2a5148e"} Nov 03 22:30:03.442362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.440698 4903 scope.go:122] "RemoveContainer" containerID="11d4f629db7cc26e0fb5b23aa47a74df47cc3dca4857b9ede2f9d8f9e2a5148e" Nov 03 22:30:03.447584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.447552 4903 generic.go:356] "Generic (PLEG): container finished" podID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerID="7192a6dc7fdb3979e0ba576c648a30501e21f424740d4001638bc977d03c7f8b" exitCode=60 Nov 03 22:30:03.447959 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.447893 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" event={"ID":"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9","Type":"ContainerDied","Data":"7192a6dc7fdb3979e0ba576c648a30501e21f424740d4001638bc977d03c7f8b"} Nov 03 22:30:03.472316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.454537 4903 scope.go:122] "RemoveContainer" containerID="7192a6dc7fdb3979e0ba576c648a30501e21f424740d4001638bc977d03c7f8b" Nov 03 22:30:03.475282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.474667 4903 generic.go:356] "Generic (PLEG): container finished" podID="d446e624-f0c0-4901-8542-a332384a8b82" containerID="731bb14e384248080056d50cff0ec336f84bac5effa7728e431a7821fa519e87" exitCode=60 Nov 03 22:30:03.475282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.474797 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" event={"ID":"d446e624-f0c0-4901-8542-a332384a8b82","Type":"ContainerDied","Data":"731bb14e384248080056d50cff0ec336f84bac5effa7728e431a7821fa519e87"} Nov 03 22:30:03.475282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.475094 4903 scope.go:122] "RemoveContainer" containerID="731bb14e384248080056d50cff0ec336f84bac5effa7728e431a7821fa519e87" Nov 03 22:30:03.478990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.477743 4903 generic.go:356] "Generic (PLEG): container finished" podID="06af2cbf-605f-41db-83be-08545aaa5b31" containerID="6417620eb565e0d15ecc82abb33dc01b5e658745877b7ae6f403ed688bdf6e13" exitCode=60 Nov 03 22:30:03.478990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.477898 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" event={"ID":"06af2cbf-605f-41db-83be-08545aaa5b31","Type":"ContainerDied","Data":"6417620eb565e0d15ecc82abb33dc01b5e658745877b7ae6f403ed688bdf6e13"} Nov 03 22:30:03.478990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.478281 4903 scope.go:122] "RemoveContainer" containerID="6417620eb565e0d15ecc82abb33dc01b5e658745877b7ae6f403ed688bdf6e13" Nov 03 22:30:03.481734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.481710 4903 generic.go:356] "Generic (PLEG): container finished" podID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerID="0cd38639c9e5591b3daf598b658352091057efed763226fb09c2d0ea90cfdc6d" exitCode=60 Nov 03 22:30:03.481932 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.481912 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" event={"ID":"21250882-c6f7-4fb4-ab40-70235609e3d4","Type":"ContainerDied","Data":"0cd38639c9e5591b3daf598b658352091057efed763226fb09c2d0ea90cfdc6d"} Nov 03 22:30:03.482426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:03.482407 4903 scope.go:122] "RemoveContainer" containerID="0cd38639c9e5591b3daf598b658352091057efed763226fb09c2d0ea90cfdc6d" Nov 03 22:30:04.502793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.502681 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" event={"ID":"5b112bfd-7ef9-4e26-954c-3627c4a3e648","Type":"ContainerStarted","Data":"1b71c901f04c86259a16d744f8289dd30abe602a0e98529a59dd52fe64277357"} Nov 03 22:30:04.505058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.505030 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" event={"ID":"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6","Type":"ContainerStarted","Data":"b70990307883cbeaef07faa630ff1d74285c6cf7144221f975dcb4a3586cc4cf"} Nov 03 22:30:04.507584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.507558 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" event={"ID":"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9","Type":"ContainerStarted","Data":"d27073c8a3e6b22b70b7dc683f42bcee250af16652f77c92c0ed3cc497b32d01"} Nov 03 22:30:04.511334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.510169 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" event={"ID":"d446e624-f0c0-4901-8542-a332384a8b82","Type":"ContainerStarted","Data":"fe6dbdee6a3eba290fcb69fae61d4ba19d30468aa58d702a265a4f7065200980"} Nov 03 22:30:04.513460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.512610 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" event={"ID":"06af2cbf-605f-41db-83be-08545aaa5b31","Type":"ContainerStarted","Data":"ea535eac360a6d98c2472c403f1128b8250a969f7e8f4e712e9c2eb0350314cd"} Nov 03 22:30:04.515214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.515152 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" event={"ID":"21250882-c6f7-4fb4-ab40-70235609e3d4","Type":"ContainerStarted","Data":"001d1f9c26b38ee0acb7acc82df400dd9a1c665c923311b5fe25709372565c45"} Nov 03 22:30:04.551179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.551120 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" podStartSLOduration=4.55110504 podStartE2EDuration="4.55110504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:04.540489817 +0000 UTC m=+1030.944430813" watchObservedRunningTime="2026-11-03 22:30:04.55110504 +0000 UTC m=+1030.955046015" Nov 03 22:30:04.579892 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.579836 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" podStartSLOduration=4.579823382 podStartE2EDuration="4.579823382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:04.577612733 +0000 UTC m=+1030.981553709" watchObservedRunningTime="2026-11-03 22:30:04.579823382 +0000 UTC m=+1030.983764371" Nov 03 22:30:04.627791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.627630 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" podStartSLOduration=4.627610154 podStartE2EDuration="4.627610154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:04.620011393 +0000 UTC m=+1031.023952364" watchObservedRunningTime="2026-11-03 22:30:04.627610154 +0000 UTC m=+1031.031551132" Nov 03 22:30:04.659482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.659422 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" podStartSLOduration=4.65940065 podStartE2EDuration="4.65940065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:04.655760241 +0000 UTC m=+1031.059701216" watchObservedRunningTime="2026-11-03 22:30:04.65940065 +0000 UTC m=+1031.063341625" Nov 03 22:30:04.695575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.695519 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" podStartSLOduration=4.695505586 podStartE2EDuration="4.695505586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:04.692251142 +0000 UTC m=+1031.096192119" watchObservedRunningTime="2026-11-03 22:30:04.695505586 +0000 UTC m=+1031.099446560" Nov 03 22:30:04.745156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:04.745090 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" podStartSLOduration=4.745067504 podStartE2EDuration="4.745067504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:04.734228013 +0000 UTC m=+1031.138168989" watchObservedRunningTime="2026-11-03 22:30:04.745067504 +0000 UTC m=+1031.149008489" Nov 03 22:30:05.533367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.533334 4903 generic.go:356] "Generic (PLEG): container finished" podID="06af2cbf-605f-41db-83be-08545aaa5b31" containerID="ea535eac360a6d98c2472c403f1128b8250a969f7e8f4e712e9c2eb0350314cd" exitCode=60 Nov 03 22:30:05.533837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.533416 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" event={"ID":"06af2cbf-605f-41db-83be-08545aaa5b31","Type":"ContainerDied","Data":"ea535eac360a6d98c2472c403f1128b8250a969f7e8f4e712e9c2eb0350314cd"} Nov 03 22:30:05.533837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.533486 4903 scope.go:122] "RemoveContainer" containerID="6417620eb565e0d15ecc82abb33dc01b5e658745877b7ae6f403ed688bdf6e13" Nov 03 22:30:05.533837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.533758 4903 scope.go:122] "RemoveContainer" containerID="ea535eac360a6d98c2472c403f1128b8250a969f7e8f4e712e9c2eb0350314cd" Nov 03 22:30:05.534221 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:05.534186 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d_odh-ai-gateway-infra(06af2cbf-605f-41db-83be-08545aaa5b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" Nov 03 22:30:05.536235 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.536204 4903 generic.go:356] "Generic (PLEG): container finished" podID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerID="001d1f9c26b38ee0acb7acc82df400dd9a1c665c923311b5fe25709372565c45" exitCode=60 Nov 03 22:30:05.536369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.536258 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" event={"ID":"21250882-c6f7-4fb4-ab40-70235609e3d4","Type":"ContainerDied","Data":"001d1f9c26b38ee0acb7acc82df400dd9a1c665c923311b5fe25709372565c45"} Nov 03 22:30:05.536664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.536648 4903 scope.go:122] "RemoveContainer" containerID="001d1f9c26b38ee0acb7acc82df400dd9a1c665c923311b5fe25709372565c45" Nov 03 22:30:05.537080 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:05.537057 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm_odh-ai-gateway-infra(21250882-c6f7-4fb4-ab40-70235609e3d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" Nov 03 22:30:05.538890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.538871 4903 generic.go:356] "Generic (PLEG): container finished" podID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerID="1b71c901f04c86259a16d744f8289dd30abe602a0e98529a59dd52fe64277357" exitCode=60 Nov 03 22:30:05.538988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.538960 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" event={"ID":"5b112bfd-7ef9-4e26-954c-3627c4a3e648","Type":"ContainerDied","Data":"1b71c901f04c86259a16d744f8289dd30abe602a0e98529a59dd52fe64277357"} Nov 03 22:30:05.539724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.539682 4903 scope.go:122] "RemoveContainer" containerID="1b71c901f04c86259a16d744f8289dd30abe602a0e98529a59dd52fe64277357" Nov 03 22:30:05.540753 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:05.540174 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-mzmlj_odh-ai-gateway-infra(5b112bfd-7ef9-4e26-954c-3627c4a3e648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" Nov 03 22:30:05.542861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.542838 4903 generic.go:356] "Generic (PLEG): container finished" podID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerID="b70990307883cbeaef07faa630ff1d74285c6cf7144221f975dcb4a3586cc4cf" exitCode=60 Nov 03 22:30:05.542972 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.542861 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" event={"ID":"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6","Type":"ContainerDied","Data":"b70990307883cbeaef07faa630ff1d74285c6cf7144221f975dcb4a3586cc4cf"} Nov 03 22:30:05.543221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.543202 4903 scope.go:122] "RemoveContainer" containerID="b70990307883cbeaef07faa630ff1d74285c6cf7144221f975dcb4a3586cc4cf" Nov 03 22:30:05.543623 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:05.543598 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s_odh-ai-gateway-infra(a2ee5b23-19dd-4f7c-83e7-470f31eb71f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" Nov 03 22:30:05.545279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.545259 4903 generic.go:356] "Generic (PLEG): container finished" podID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerID="d27073c8a3e6b22b70b7dc683f42bcee250af16652f77c92c0ed3cc497b32d01" exitCode=60 Nov 03 22:30:05.545378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.545321 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" event={"ID":"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9","Type":"ContainerDied","Data":"d27073c8a3e6b22b70b7dc683f42bcee250af16652f77c92c0ed3cc497b32d01"} Nov 03 22:30:05.545720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.545700 4903 scope.go:122] "RemoveContainer" containerID="d27073c8a3e6b22b70b7dc683f42bcee250af16652f77c92c0ed3cc497b32d01" Nov 03 22:30:05.546094 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:05.546062 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj_odh-ai-gateway-infra(8c583acd-e85b-4b5c-83d8-0b5e7774dbc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" Nov 03 22:30:05.547769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.547745 4903 generic.go:356] "Generic (PLEG): container finished" podID="d446e624-f0c0-4901-8542-a332384a8b82" containerID="fe6dbdee6a3eba290fcb69fae61d4ba19d30468aa58d702a265a4f7065200980" exitCode=60 Nov 03 22:30:05.547887 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.547780 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" event={"ID":"d446e624-f0c0-4901-8542-a332384a8b82","Type":"ContainerDied","Data":"fe6dbdee6a3eba290fcb69fae61d4ba19d30468aa58d702a265a4f7065200980"} Nov 03 22:30:05.548066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.548051 4903 scope.go:122] "RemoveContainer" containerID="fe6dbdee6a3eba290fcb69fae61d4ba19d30468aa58d702a265a4f7065200980" Nov 03 22:30:05.548381 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:05.548359 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq_odh-ai-gateway-infra(d446e624-f0c0-4901-8542-a332384a8b82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" podUID="d446e624-f0c0-4901-8542-a332384a8b82" Nov 03 22:30:05.554673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.554651 4903 scope.go:122] "RemoveContainer" containerID="0cd38639c9e5591b3daf598b658352091057efed763226fb09c2d0ea90cfdc6d" Nov 03 22:30:05.571502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.571475 4903 scope.go:122] "RemoveContainer" containerID="4b0c2a1298efd0517d542e15f5454309af8c0ad2e2268735c3bf73016ee27493" Nov 03 22:30:05.589138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.589112 4903 scope.go:122] "RemoveContainer" containerID="11d4f629db7cc26e0fb5b23aa47a74df47cc3dca4857b9ede2f9d8f9e2a5148e" Nov 03 22:30:05.606521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.606473 4903 scope.go:122] "RemoveContainer" containerID="7192a6dc7fdb3979e0ba576c648a30501e21f424740d4001638bc977d03c7f8b" Nov 03 22:30:05.622820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:05.622799 4903 scope.go:122] "RemoveContainer" containerID="731bb14e384248080056d50cff0ec336f84bac5effa7728e431a7821fa519e87" Nov 03 22:30:06.563833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:06.563747 4903 scope.go:122] "RemoveContainer" containerID="1b71c901f04c86259a16d744f8289dd30abe602a0e98529a59dd52fe64277357" Nov 03 22:30:06.564282 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:06.564176 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895750-mzmlj_odh-ai-gateway-infra(5b112bfd-7ef9-4e26-954c-3627c4a3e648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" Nov 03 22:30:06.566023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:06.565999 4903 scope.go:122] "RemoveContainer" containerID="b70990307883cbeaef07faa630ff1d74285c6cf7144221f975dcb4a3586cc4cf" Nov 03 22:30:06.566375 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:06.566353 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s_odh-ai-gateway-infra(a2ee5b23-19dd-4f7c-83e7-470f31eb71f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" Nov 03 22:30:06.567695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:06.567677 4903 scope.go:122] "RemoveContainer" containerID="d27073c8a3e6b22b70b7dc683f42bcee250af16652f77c92c0ed3cc497b32d01" Nov 03 22:30:06.567996 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:06.567973 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj_odh-ai-gateway-infra(8c583acd-e85b-4b5c-83d8-0b5e7774dbc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" Nov 03 22:30:06.569613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:06.569592 4903 scope.go:122] "RemoveContainer" containerID="fe6dbdee6a3eba290fcb69fae61d4ba19d30468aa58d702a265a4f7065200980" Nov 03 22:30:06.569918 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:06.569891 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq_odh-ai-gateway-infra(d446e624-f0c0-4901-8542-a332384a8b82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" podUID="d446e624-f0c0-4901-8542-a332384a8b82" Nov 03 22:30:06.571191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:06.571172 4903 scope.go:122] "RemoveContainer" containerID="ea535eac360a6d98c2472c403f1128b8250a969f7e8f4e712e9c2eb0350314cd" Nov 03 22:30:06.571509 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:06.571490 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d_odh-ai-gateway-infra(06af2cbf-605f-41db-83be-08545aaa5b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" Nov 03 22:30:06.572888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:06.572871 4903 scope.go:122] "RemoveContainer" containerID="001d1f9c26b38ee0acb7acc82df400dd9a1c665c923311b5fe25709372565c45" Nov 03 22:30:06.573111 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:06.573094 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm_odh-ai-gateway-infra(21250882-c6f7-4fb4-ab40-70235609e3d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" Nov 03 22:30:13.041417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:13.041376 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5"] Nov 03 22:30:13.050434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:13.050406 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987e6ab4-5ptx5"] Nov 03 22:30:13.738954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:13.738919 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c043b1b2-b329-4a63-b92f-62a82c1841cf" path="/var/lib/kubelet/pods/c043b1b2-b329-4a63-b92f-62a82c1841cf/volumes" Nov 03 22:30:14.194076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.194043 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd"] Nov 03 22:30:14.200517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.200492 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.237210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.237166 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.237417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.237242 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.237417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.237350 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.237417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.237408 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.237570 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.237437 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.237570 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.237517 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.237762 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.237655 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.237762 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.237720 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z58ct\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-kube-api-access-z58ct\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.237966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.237935 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd"] Nov 03 22:30:14.238035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.238009 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.339840 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.339804 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.339860 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z58ct\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-kube-api-access-z58ct\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.339927 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340070 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340120 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340186 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340235 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340260 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340382 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340510 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-credential-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340592 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-certs\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340666 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-socket\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340677 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-data\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.340829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.340805 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istiod-ca-cert\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.342572 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.342543 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-envoy\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.342800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.342779 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-podinfo\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.373808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.373727 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-token\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.453828 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.443089 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z58ct\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-kube-api-access-z58ct\") pod \"e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.544341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.540225 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:14.893673 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:30:14.893640 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa58306f_409e_4dfb_8723_2f1ae4e66d18.slice/crio-fa9ed72ba38451c2376ea54d68370914d72335307fa13e7480dfbde2f6537b6d WatchSource:0}: Error finding container fa9ed72ba38451c2376ea54d68370914d72335307fa13e7480dfbde2f6537b6d: Status 404 returned error can't find the container with id fa9ed72ba38451c2376ea54d68370914d72335307fa13e7480dfbde2f6537b6d Nov 03 22:30:14.895677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.895652 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd"] Nov 03 22:30:14.896639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.896601 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:14.896710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.896693 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:14.896755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:14.896744 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:30:15.680795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:15.680756 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" event={"ID":"aa58306f-409e-4dfb-8723-2f1ae4e66d18","Type":"ContainerStarted","Data":"5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd"} Nov 03 22:30:15.681208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:15.680816 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" event={"ID":"aa58306f-409e-4dfb-8723-2f1ae4e66d18","Type":"ContainerStarted","Data":"fa9ed72ba38451c2376ea54d68370914d72335307fa13e7480dfbde2f6537b6d"} Nov 03 22:30:15.726383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:15.726353 4903 scope.go:122] "RemoveContainer" containerID="fe6dbdee6a3eba290fcb69fae61d4ba19d30468aa58d702a265a4f7065200980" Nov 03 22:30:15.770089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:15.770015 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" podStartSLOduration=1.7699952589999999 podStartE2EDuration="1.769995259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:15.732604381 +0000 UTC m=+1042.136545356" watchObservedRunningTime="2026-11-03 22:30:15.769995259 +0000 UTC m=+1042.173936235" Nov 03 22:30:16.543580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:16.543492 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:16.543919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:16.543897 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:30:16.544077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:16.544051 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 22:30:16.692358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:16.692319 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" event={"ID":"d446e624-f0c0-4901-8542-a332384a8b82","Type":"ContainerStarted","Data":"913d1bad1605d77025c0a886957211dc9d385470936d4f74051a8b26f293b7f7"} Nov 03 22:30:16.726220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:16.726192 4903 scope.go:122] "RemoveContainer" containerID="ea535eac360a6d98c2472c403f1128b8250a969f7e8f4e712e9c2eb0350314cd" Nov 03 22:30:17.562376 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.545624 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:30:17.562376 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.545683 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 22:30:17.707355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.707313 4903 generic.go:356] "Generic (PLEG): container finished" podID="d446e624-f0c0-4901-8542-a332384a8b82" containerID="913d1bad1605d77025c0a886957211dc9d385470936d4f74051a8b26f293b7f7" exitCode=60 Nov 03 22:30:17.707845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.707459 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" event={"ID":"d446e624-f0c0-4901-8542-a332384a8b82","Type":"ContainerDied","Data":"913d1bad1605d77025c0a886957211dc9d385470936d4f74051a8b26f293b7f7"} Nov 03 22:30:17.707845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.707557 4903 scope.go:122] "RemoveContainer" containerID="fe6dbdee6a3eba290fcb69fae61d4ba19d30468aa58d702a265a4f7065200980" Nov 03 22:30:17.708213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.708192 4903 scope.go:122] "RemoveContainer" containerID="913d1bad1605d77025c0a886957211dc9d385470936d4f74051a8b26f293b7f7" Nov 03 22:30:17.708911 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:17.708882 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq_odh-ai-gateway-infra(d446e624-f0c0-4901-8542-a332384a8b82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" podUID="d446e624-f0c0-4901-8542-a332384a8b82" Nov 03 22:30:17.722403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.722365 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" event={"ID":"06af2cbf-605f-41db-83be-08545aaa5b31","Type":"ContainerStarted","Data":"4d20a085282443d6f8d1b57b51adce77a859366dc5a96618efef26da3e0d081b"} Nov 03 22:30:17.726336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.726315 4903 scope.go:122] "RemoveContainer" containerID="1b71c901f04c86259a16d744f8289dd30abe602a0e98529a59dd52fe64277357" Nov 03 22:30:17.726470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.726430 4903 scope.go:122] "RemoveContainer" containerID="001d1f9c26b38ee0acb7acc82df400dd9a1c665c923311b5fe25709372565c45" Nov 03 22:30:17.726644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.726624 4903 scope.go:122] "RemoveContainer" containerID="d27073c8a3e6b22b70b7dc683f42bcee250af16652f77c92c0ed3cc497b32d01" Nov 03 22:30:17.773863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:17.773814 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq"] Nov 03 22:30:18.036366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.036331 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz"] Nov 03 22:30:18.066368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.063700 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl"] Nov 03 22:30:18.094736 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.094708 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2020770c-wvsjz"] Nov 03 22:30:18.101917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.101884 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b0790a-dpgjl"] Nov 03 22:30:18.402833 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:18.402794 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:30:18.566054 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.565901 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:30:18.566054 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.565962 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 22:30:18.726181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.726145 4903 scope.go:122] "RemoveContainer" containerID="b70990307883cbeaef07faa630ff1d74285c6cf7144221f975dcb4a3586cc4cf" Nov 03 22:30:18.736914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.736835 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" event={"ID":"5b112bfd-7ef9-4e26-954c-3627c4a3e648","Type":"ContainerStarted","Data":"316a7db7f20f3e8a2b44fd46aa0119af41079f1463ab46196f120d53dff9415b"} Nov 03 22:30:18.739433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.739400 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" event={"ID":"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9","Type":"ContainerStarted","Data":"cbc699f3aeaf0dbf7890dd2000624b6cc55e43f18353e43638dbf6d1ffacb8a7"} Nov 03 22:30:18.775664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.775626 4903 generic.go:356] "Generic (PLEG): container finished" podID="06af2cbf-605f-41db-83be-08545aaa5b31" containerID="4d20a085282443d6f8d1b57b51adce77a859366dc5a96618efef26da3e0d081b" exitCode=60 Nov 03 22:30:18.775818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.775775 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" event={"ID":"06af2cbf-605f-41db-83be-08545aaa5b31","Type":"ContainerDied","Data":"4d20a085282443d6f8d1b57b51adce77a859366dc5a96618efef26da3e0d081b"} Nov 03 22:30:18.775882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.775826 4903 scope.go:122] "RemoveContainer" containerID="ea535eac360a6d98c2472c403f1128b8250a969f7e8f4e712e9c2eb0350314cd" Nov 03 22:30:18.776287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.776265 4903 scope.go:122] "RemoveContainer" containerID="4d20a085282443d6f8d1b57b51adce77a859366dc5a96618efef26da3e0d081b" Nov 03 22:30:18.777458 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:18.777424 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d_odh-ai-gateway-infra(06af2cbf-605f-41db-83be-08545aaa5b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" Nov 03 22:30:18.792511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.792171 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" event={"ID":"21250882-c6f7-4fb4-ab40-70235609e3d4","Type":"ContainerStarted","Data":"84afdd7570bb92fe9d29f0df08d21672d521101582aaf1281163d54285574305"} Nov 03 22:30:18.830973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:18.830943 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d"] Nov 03 22:30:19.075855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.075788 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n"] Nov 03 22:30:19.091620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.091586 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75577660-78s6n"] Nov 03 22:30:19.175038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.175010 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:19.290485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.289981 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d446e624-f0c0-4901-8542-a332384a8b82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446e624-f0c0-4901-8542-a332384a8b82-maas-api-service-ca\") pod \"d446e624-f0c0-4901-8542-a332384a8b82\" (UID: \"d446e624-f0c0-4901-8542-a332384a8b82\") " Nov 03 22:30:19.290485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.290188 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d446e624-f0c0-4901-8542-a332384a8b82-kube-api-access-nz8ls\" (UniqueName: \"kubernetes.io/projected/d446e624-f0c0-4901-8542-a332384a8b82-kube-api-access-nz8ls\") pod \"d446e624-f0c0-4901-8542-a332384a8b82\" (UID: \"d446e624-f0c0-4901-8542-a332384a8b82\") " Nov 03 22:30:19.290703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.290489 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d446e624-f0c0-4901-8542-a332384a8b82-maas-api-service-ca" pod "d446e624-f0c0-4901-8542-a332384a8b82" (UID: "d446e624-f0c0-4901-8542-a332384a8b82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:19.290954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.290929 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446e624-f0c0-4901-8542-a332384a8b82-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:19.292803 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.292775 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d446e624-f0c0-4901-8542-a332384a8b82-kube-api-access-nz8ls" pod "d446e624-f0c0-4901-8542-a332384a8b82" (UID: "d446e624-f0c0-4901-8542-a332384a8b82"). InnerVolumeSpecName "kube-api-access-nz8ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:19.393127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.393099 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz8ls\" (UniqueName: \"kubernetes.io/projected/d446e624-f0c0-4901-8542-a332384a8b82-kube-api-access-nz8ls\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:19.560374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.548652 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:30:19.560374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.548712 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 22:30:19.736513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.736477 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe7a4d4-ec2f-4024-ac21-03e677210362" path="/var/lib/kubelet/pods/3fe7a4d4-ec2f-4024-ac21-03e677210362/volumes" Nov 03 22:30:19.737172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.737153 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0" path="/var/lib/kubelet/pods/bf2d660a-a0b6-43f7-bd95-425c8b7fe7b0/volumes" Nov 03 22:30:19.737731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.737708 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4e0be2-8036-422b-8be4-0b904c0e0334" path="/var/lib/kubelet/pods/ff4e0be2-8036-422b-8be4-0b904c0e0334/volumes" Nov 03 22:30:19.797185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.797148 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj"] Nov 03 22:30:19.817001 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.816907 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" event={"ID":"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6","Type":"ContainerStarted","Data":"6eaf22c2c3bb45688720e729142a09a0d2a7bd9e33f98d596abcef1f6934fb65"} Nov 03 22:30:19.829362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.825482 4903 generic.go:356] "Generic (PLEG): container finished" podID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerID="cbc699f3aeaf0dbf7890dd2000624b6cc55e43f18353e43638dbf6d1ffacb8a7" exitCode=60 Nov 03 22:30:19.829362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.825630 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" event={"ID":"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9","Type":"ContainerDied","Data":"cbc699f3aeaf0dbf7890dd2000624b6cc55e43f18353e43638dbf6d1ffacb8a7"} Nov 03 22:30:19.829362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.825685 4903 scope.go:122] "RemoveContainer" containerID="d27073c8a3e6b22b70b7dc683f42bcee250af16652f77c92c0ed3cc497b32d01" Nov 03 22:30:19.829362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.826196 4903 scope.go:122] "RemoveContainer" containerID="cbc699f3aeaf0dbf7890dd2000624b6cc55e43f18353e43638dbf6d1ffacb8a7" Nov 03 22:30:19.829362 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:19.826664 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj_odh-ai-gateway-infra(8c583acd-e85b-4b5c-83d8-0b5e7774dbc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" Nov 03 22:30:19.852410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.839788 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" event={"ID":"d446e624-f0c0-4901-8542-a332384a8b82","Type":"ContainerDied","Data":"10fa75358af59046e69b8eaf48594da559182f34ce9b1b37dd67ebf7ea8bcbdb"} Nov 03 22:30:19.852410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.839938 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq" Nov 03 22:30:19.977669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:19.977629 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj"] Nov 03 22:30:20.010327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.005998 4903 generic.go:356] "Generic (PLEG): container finished" podID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerID="84afdd7570bb92fe9d29f0df08d21672d521101582aaf1281163d54285574305" exitCode=60 Nov 03 22:30:20.010327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.006118 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" event={"ID":"21250882-c6f7-4fb4-ab40-70235609e3d4","Type":"ContainerDied","Data":"84afdd7570bb92fe9d29f0df08d21672d521101582aaf1281163d54285574305"} Nov 03 22:30:20.010327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.007657 4903 scope.go:122] "RemoveContainer" containerID="84afdd7570bb92fe9d29f0df08d21672d521101582aaf1281163d54285574305" Nov 03 22:30:20.010327 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:20.008811 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm_odh-ai-gateway-infra(21250882-c6f7-4fb4-ab40-70235609e3d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" Nov 03 22:30:20.010705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.010680 4903 generic.go:356] "Generic (PLEG): container finished" podID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerID="316a7db7f20f3e8a2b44fd46aa0119af41079f1463ab46196f120d53dff9415b" exitCode=60 Nov 03 22:30:20.010960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.010942 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" event={"ID":"5b112bfd-7ef9-4e26-954c-3627c4a3e648","Type":"ContainerDied","Data":"316a7db7f20f3e8a2b44fd46aa0119af41079f1463ab46196f120d53dff9415b"} Nov 03 22:30:20.031665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.031575 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm"] Nov 03 22:30:20.085075 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.081804 4903 scope.go:122] "RemoveContainer" containerID="913d1bad1605d77025c0a886957211dc9d385470936d4f74051a8b26f293b7f7" Nov 03 22:30:20.102265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.102236 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq"] Nov 03 22:30:20.102435 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.102286 4903 scope.go:122] "RemoveContainer" containerID="001d1f9c26b38ee0acb7acc82df400dd9a1c665c923311b5fe25709372565c45" Nov 03 22:30:20.106230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.105925 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895750-qkldq"] Nov 03 22:30:20.129548 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:20.129514 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ee5b23_19dd_4f7c_83e7_470f31eb71f6.slice/crio-6eaf22c2c3bb45688720e729142a09a0d2a7bd9e33f98d596abcef1f6934fb65.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ee5b23_19dd_4f7c_83e7_470f31eb71f6.slice/crio-conmon-6eaf22c2c3bb45688720e729142a09a0d2a7bd9e33f98d596abcef1f6934fb65.scope\": RecentStats: unable to find data in memory cache]" Nov 03 22:30:20.206962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.203510 4903 scope.go:122] "RemoveContainer" containerID="1b71c901f04c86259a16d744f8289dd30abe602a0e98529a59dd52fe64277357" Nov 03 22:30:20.389389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.389365 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:20.391981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.391959 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:20.498522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.498482 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06af2cbf-605f-41db-83be-08545aaa5b31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06af2cbf-605f-41db-83be-08545aaa5b31-maas-api-service-ca\") pod \"06af2cbf-605f-41db-83be-08545aaa5b31\" (UID: \"06af2cbf-605f-41db-83be-08545aaa5b31\") " Nov 03 22:30:20.499633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.499599 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06af2cbf-605f-41db-83be-08545aaa5b31-maas-api-service-ca" pod "06af2cbf-605f-41db-83be-08545aaa5b31" (UID: "06af2cbf-605f-41db-83be-08545aaa5b31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:20.501333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.501303 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b112bfd-7ef9-4e26-954c-3627c4a3e648-kube-api-access-m5fpn\" (UniqueName: \"kubernetes.io/projected/5b112bfd-7ef9-4e26-954c-3627c4a3e648-kube-api-access-m5fpn\") pod \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\" (UID: \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\") " Nov 03 22:30:20.501458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.501354 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b112bfd-7ef9-4e26-954c-3627c4a3e648-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b112bfd-7ef9-4e26-954c-3627c4a3e648-maas-api-service-ca\") pod \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\" (UID: \"5b112bfd-7ef9-4e26-954c-3627c4a3e648\") " Nov 03 22:30:20.501519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.501472 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06af2cbf-605f-41db-83be-08545aaa5b31-kube-api-access-c89c6\" (UniqueName: \"kubernetes.io/projected/06af2cbf-605f-41db-83be-08545aaa5b31-kube-api-access-c89c6\") pod \"06af2cbf-605f-41db-83be-08545aaa5b31\" (UID: \"06af2cbf-605f-41db-83be-08545aaa5b31\") " Nov 03 22:30:20.502077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.502056 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06af2cbf-605f-41db-83be-08545aaa5b31-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:20.505052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.505028 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b112bfd-7ef9-4e26-954c-3627c4a3e648-kube-api-access-m5fpn" pod "5b112bfd-7ef9-4e26-954c-3627c4a3e648" (UID: "5b112bfd-7ef9-4e26-954c-3627c4a3e648"). InnerVolumeSpecName "kube-api-access-m5fpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:20.505548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.505523 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b112bfd-7ef9-4e26-954c-3627c4a3e648-maas-api-service-ca" pod "5b112bfd-7ef9-4e26-954c-3627c4a3e648" (UID: "5b112bfd-7ef9-4e26-954c-3627c4a3e648"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:20.520517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.520484 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06af2cbf-605f-41db-83be-08545aaa5b31-kube-api-access-c89c6" pod "06af2cbf-605f-41db-83be-08545aaa5b31" (UID: "06af2cbf-605f-41db-83be-08545aaa5b31"). InnerVolumeSpecName "kube-api-access-c89c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:20.554357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.552453 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:30:20.554357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.552511 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 22:30:20.603876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.603792 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b112bfd-7ef9-4e26-954c-3627c4a3e648-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:20.603876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.603822 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c89c6\" (UniqueName: \"kubernetes.io/projected/06af2cbf-605f-41db-83be-08545aaa5b31-kube-api-access-c89c6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:20.603876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.603832 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5fpn\" (UniqueName: \"kubernetes.io/projected/5b112bfd-7ef9-4e26-954c-3627c4a3e648-kube-api-access-m5fpn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:20.878163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:20.878072 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s"] Nov 03 22:30:21.035001 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.034973 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" Nov 03 22:30:21.036800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.035095 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d" event={"ID":"06af2cbf-605f-41db-83be-08545aaa5b31","Type":"ContainerDied","Data":"d06090fc2126c7df2a80912f3176b350edc8030d56ec849f4baaadc0708c9004"} Nov 03 22:30:21.036960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.036940 4903 scope.go:122] "RemoveContainer" containerID="4d20a085282443d6f8d1b57b51adce77a859366dc5a96618efef26da3e0d081b" Nov 03 22:30:21.044318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.043206 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" event={"ID":"5b112bfd-7ef9-4e26-954c-3627c4a3e648","Type":"ContainerDied","Data":"8bc5336108b46388312b6faba6d4c2736ebfe8fb2d523e9e968dfd6090b53336"} Nov 03 22:30:21.044318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.043363 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj" Nov 03 22:30:21.059486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.059454 4903 generic.go:356] "Generic (PLEG): container finished" podID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerID="6eaf22c2c3bb45688720e729142a09a0d2a7bd9e33f98d596abcef1f6934fb65" exitCode=60 Nov 03 22:30:21.059656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.059524 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" event={"ID":"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6","Type":"ContainerDied","Data":"6eaf22c2c3bb45688720e729142a09a0d2a7bd9e33f98d596abcef1f6934fb65"} Nov 03 22:30:21.158882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.158850 4903 scope.go:122] "RemoveContainer" containerID="316a7db7f20f3e8a2b44fd46aa0119af41079f1463ab46196f120d53dff9415b" Nov 03 22:30:21.203903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.203005 4903 scope.go:122] "RemoveContainer" containerID="b70990307883cbeaef07faa630ff1d74285c6cf7144221f975dcb4a3586cc4cf" Nov 03 22:30:21.210372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.209687 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj"] Nov 03 22:30:21.219050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.218285 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895750-mzmlj"] Nov 03 22:30:21.306249 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.302811 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d"] Nov 03 22:30:21.328840 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.327540 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-88a4d9-29895750-swl2d"] Nov 03 22:30:21.523538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.523503 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:21.542689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.540964 4903 patch_prober.go:29] interesting pod/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 22:30:21.542689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.541048 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 22:30:21.548333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.547390 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:21.549466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.549440 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:21.677754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.677720 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-kube-api-access-plj9v\" (UniqueName: \"kubernetes.io/projected/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-kube-api-access-plj9v\") pod \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\" (UID: \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\") " Nov 03 22:30:21.677955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.677802 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21250882-c6f7-4fb4-ab40-70235609e3d4-kube-api-access-w442f\" (UniqueName: \"kubernetes.io/projected/21250882-c6f7-4fb4-ab40-70235609e3d4-kube-api-access-w442f\") pod \"21250882-c6f7-4fb4-ab40-70235609e3d4\" (UID: \"21250882-c6f7-4fb4-ab40-70235609e3d4\") " Nov 03 22:30:21.677955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.677880 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-maas-api-service-ca\") pod \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\" (UID: \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\") " Nov 03 22:30:21.678070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.678057 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-maas-api-service-ca\") pod \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\" (UID: \"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6\") " Nov 03 22:30:21.678125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.678104 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21250882-c6f7-4fb4-ab40-70235609e3d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21250882-c6f7-4fb4-ab40-70235609e3d4-maas-api-service-ca\") pod \"21250882-c6f7-4fb4-ab40-70235609e3d4\" (UID: \"21250882-c6f7-4fb4-ab40-70235609e3d4\") " Nov 03 22:30:21.679002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.678965 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-maas-api-service-ca" pod "8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" (UID: "8c583acd-e85b-4b5c-83d8-0b5e7774dbc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:21.679269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.679058 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21250882-c6f7-4fb4-ab40-70235609e3d4-maas-api-service-ca" pod "21250882-c6f7-4fb4-ab40-70235609e3d4" (UID: "21250882-c6f7-4fb4-ab40-70235609e3d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:21.679372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.679216 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-maas-api-service-ca" pod "a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" (UID: "a2ee5b23-19dd-4f7c-83e7-470f31eb71f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:21.679701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.679681 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-kube-api-access-cg5r5\" (UniqueName: \"kubernetes.io/projected/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-kube-api-access-cg5r5\") pod \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\" (UID: \"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9\") " Nov 03 22:30:21.680748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.680718 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:21.680853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.680842 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:21.680963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.680954 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21250882-c6f7-4fb4-ab40-70235609e3d4-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:21.681282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.681268 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21250882-c6f7-4fb4-ab40-70235609e3d4-kube-api-access-w442f" pod "21250882-c6f7-4fb4-ab40-70235609e3d4" (UID: "21250882-c6f7-4fb4-ab40-70235609e3d4"). InnerVolumeSpecName "kube-api-access-w442f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:21.682237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.682207 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-kube-api-access-plj9v" pod "a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" (UID: "a2ee5b23-19dd-4f7c-83e7-470f31eb71f6"). InnerVolumeSpecName "kube-api-access-plj9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:21.682936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.682914 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-kube-api-access-cg5r5" pod "8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" (UID: "8c583acd-e85b-4b5c-83d8-0b5e7774dbc9"). InnerVolumeSpecName "kube-api-access-cg5r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:21.753310 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.753261 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" path="/var/lib/kubelet/pods/06af2cbf-605f-41db-83be-08545aaa5b31/volumes" Nov 03 22:30:21.754217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.754196 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" path="/var/lib/kubelet/pods/5b112bfd-7ef9-4e26-954c-3627c4a3e648/volumes" Nov 03 22:30:21.754946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.754927 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d446e624-f0c0-4901-8542-a332384a8b82" path="/var/lib/kubelet/pods/d446e624-f0c0-4901-8542-a332384a8b82/volumes" Nov 03 22:30:21.783014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.782987 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg5r5\" (UniqueName: \"kubernetes.io/projected/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9-kube-api-access-cg5r5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:21.783198 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.783173 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plj9v\" (UniqueName: \"kubernetes.io/projected/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6-kube-api-access-plj9v\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:21.783198 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:21.783200 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w442f\" (UniqueName: \"kubernetes.io/projected/21250882-c6f7-4fb4-ab40-70235609e3d4-kube-api-access-w442f\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:22.075527 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.075447 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" Nov 03 22:30:22.076030 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.075446 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s" event={"ID":"a2ee5b23-19dd-4f7c-83e7-470f31eb71f6","Type":"ContainerDied","Data":"0f2d41aed26039b75ea2972877d0a96fc8e8a4d8f6a16348ed58112e71781f5f"} Nov 03 22:30:22.076131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.076078 4903 scope.go:122] "RemoveContainer" containerID="6eaf22c2c3bb45688720e729142a09a0d2a7bd9e33f98d596abcef1f6934fb65" Nov 03 22:30:22.078852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.078092 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" event={"ID":"8c583acd-e85b-4b5c-83d8-0b5e7774dbc9","Type":"ContainerDied","Data":"99d8f86ea7f5e5c35fe2acdb5f57ac6e33929e409e352f8d859f81edbc8430dc"} Nov 03 22:30:22.078852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.078192 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj" Nov 03 22:30:22.113637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.113599 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" event={"ID":"21250882-c6f7-4fb4-ab40-70235609e3d4","Type":"ContainerDied","Data":"2bcaf5630443103024c7b4fffc8d4dcf07266b01961dd5068298a98f4ed8057e"} Nov 03 22:30:22.114053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.113980 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm" Nov 03 22:30:22.132883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.132424 4903 scope.go:122] "RemoveContainer" containerID="cbc699f3aeaf0dbf7890dd2000624b6cc55e43f18353e43638dbf6d1ffacb8a7" Nov 03 22:30:22.157258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.157229 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s"] Nov 03 22:30:22.160337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.160286 4903 scope.go:122] "RemoveContainer" containerID="84afdd7570bb92fe9d29f0df08d21672d521101582aaf1281163d54285574305" Nov 03 22:30:22.167382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.166149 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895750-qvw4s"] Nov 03 22:30:22.197932 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.197901 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj"] Nov 03 22:30:22.204078 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.204049 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-51bfa2e1-29895750-ffmcj"] Nov 03 22:30:22.232275 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.232239 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm"] Nov 03 22:30:22.240415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.240384 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895750-42rrm"] Nov 03 22:30:22.590125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.590096 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:22.590566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.590541 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:22.591154 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:22.591136 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 22:30:23.046117 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:23.046068 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb"] Nov 03 22:30:23.056437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:23.056399 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb0e211a-s8ggb"] Nov 03 22:30:23.739703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:23.739667 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" path="/var/lib/kubelet/pods/21250882-c6f7-4fb4-ab40-70235609e3d4/volumes" Nov 03 22:30:23.740340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:23.740286 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" path="/var/lib/kubelet/pods/8c583acd-e85b-4b5c-83d8-0b5e7774dbc9/volumes" Nov 03 22:30:23.740893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:23.740877 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945da303-35e3-4f29-b55c-b46150e7aa95" path="/var/lib/kubelet/pods/945da303-35e3-4f29-b55c-b46150e7aa95/volumes" Nov 03 22:30:23.741809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:23.741790 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" path="/var/lib/kubelet/pods/a2ee5b23-19dd-4f7c-83e7-470f31eb71f6/volumes" Nov 03 22:30:26.690725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.690690 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q"] Nov 03 22:30:26.691419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691397 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d446e624-f0c0-4901-8542-a332384a8b82" containerName="cleanup" Nov 03 22:30:26.691475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691421 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d446e624-f0c0-4901-8542-a332384a8b82" containerName="cleanup" Nov 03 22:30:26.691475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691431 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerName="cleanup" Nov 03 22:30:26.691475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691436 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerName="cleanup" Nov 03 22:30:26.691475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691445 4903 cpu_manager.go:414] "containerMap: removing container" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerName="cleanup" Nov 03 22:30:26.691475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691451 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerName="cleanup" Nov 03 22:30:26.691475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691461 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerName="cleanup" Nov 03 22:30:26.691475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691465 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerName="cleanup" Nov 03 22:30:26.691475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691471 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerName="cleanup" Nov 03 22:30:26.691475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691475 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691489 4903 cpu_manager.go:414] "containerMap: removing container" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691493 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691506 4903 cpu_manager.go:414] "containerMap: removing container" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691511 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691518 4903 cpu_manager.go:414] "containerMap: removing container" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691523 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691529 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691533 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691546 4903 cpu_manager.go:414] "containerMap: removing container" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691551 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691570 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691575 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691584 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d446e624-f0c0-4901-8542-a332384a8b82" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691589 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d446e624-f0c0-4901-8542-a332384a8b82" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691601 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691605 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691723 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691732 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d446e624-f0c0-4901-8542-a332384a8b82" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691739 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerName="cleanup" Nov 03 22:30:26.691740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691746 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691754 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691760 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691765 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d446e624-f0c0-4901-8542-a332384a8b82" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691769 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691774 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691779 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691784 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691791 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="21250882-c6f7-4fb4-ab40-70235609e3d4" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691796 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerName="cleanup" Nov 03 22:30:26.692287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.691803 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" containerName="cleanup" Nov 03 22:30:26.696982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.696959 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:26.729454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.729419 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q"] Nov 03 22:30:26.730962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.730940 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:30:26.731046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.731003 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:30:26.731138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.731125 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:30:26.731203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.731192 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:30:26.882732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.882701 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fb51a2-cdk2q\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:26.882914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.882762 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fb51a2-cdk2q\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:26.882957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.882933 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjn4\" (UniqueName: \"kubernetes.io/projected/818eafd8-baaa-4905-b3cc-237c00fdcb73-kube-api-access-6hjn4\") pod \"maas-api-revoke-sub-14fb51a2-cdk2q\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:26.984841 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.984745 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fb51a2-cdk2q\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:26.984841 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.984818 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fb51a2-cdk2q\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:26.985075 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.984929 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjn4\" (UniqueName: \"kubernetes.io/projected/818eafd8-baaa-4905-b3cc-237c00fdcb73-kube-api-access-6hjn4\") pod \"maas-api-revoke-sub-14fb51a2-cdk2q\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:26.985472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.985447 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fb51a2-cdk2q\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:26.987247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:26.987218 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fb51a2-cdk2q\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:27.070333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:27.070271 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjn4\" (UniqueName: \"kubernetes.io/projected/818eafd8-baaa-4905-b3cc-237c00fdcb73-kube-api-access-6hjn4\") pod \"maas-api-revoke-sub-14fb51a2-cdk2q\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:27.189964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:27.189929 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl"] Nov 03 22:30:27.196124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:27.196087 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2"] Nov 03 22:30:27.202261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:27.202233 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1fb9e2-nlcsl"] Nov 03 22:30:27.207021 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:27.206995 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78ba40e6-h9qb2"] Nov 03 22:30:27.313013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:27.312935 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:27.632926 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:30:27.632899 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818eafd8_baaa_4905_b3cc_237c00fdcb73.slice/crio-63b37bc1b5ca5a429cebee5e0ea4f865fc28538a9500913964cc25c20af4ecd4 WatchSource:0}: Error finding container 63b37bc1b5ca5a429cebee5e0ea4f865fc28538a9500913964cc25c20af4ecd4: Status 404 returned error can't find the container with id 63b37bc1b5ca5a429cebee5e0ea4f865fc28538a9500913964cc25c20af4ecd4 Nov 03 22:30:27.638409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:27.638099 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q"] Nov 03 22:30:27.742580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:27.742526 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be1f7db-bf6a-4455-9c37-b10b499e5b8d" path="/var/lib/kubelet/pods/6be1f7db-bf6a-4455-9c37-b10b499e5b8d/volumes" Nov 03 22:30:27.743236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:27.743215 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc17242f-1ea8-421a-8fc8-c6f784447536" path="/var/lib/kubelet/pods/bc17242f-1ea8-421a-8fc8-c6f784447536/volumes" Nov 03 22:30:28.182478 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:28.182442 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" event={"ID":"818eafd8-baaa-4905-b3cc-237c00fdcb73","Type":"ContainerStarted","Data":"f5d97d89fcb917ea9e8b0eeeb8f2b0d20c087cef6217422c7040120e231a721f"} Nov 03 22:30:28.182683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:28.182496 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" event={"ID":"818eafd8-baaa-4905-b3cc-237c00fdcb73","Type":"ContainerStarted","Data":"63b37bc1b5ca5a429cebee5e0ea4f865fc28538a9500913964cc25c20af4ecd4"} Nov 03 22:30:28.229013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:28.228881 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" podStartSLOduration=2.228864385 podStartE2EDuration="2.228864385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:28.228205228 +0000 UTC m=+1054.632146206" watchObservedRunningTime="2026-11-03 22:30:28.228864385 +0000 UTC m=+1054.632805355" Nov 03 22:30:29.195943 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:29.195902 4903 generic.go:356] "Generic (PLEG): container finished" podID="818eafd8-baaa-4905-b3cc-237c00fdcb73" containerID="f5d97d89fcb917ea9e8b0eeeb8f2b0d20c087cef6217422c7040120e231a721f" exitCode=0 Nov 03 22:30:29.196440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:29.195984 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" event={"ID":"818eafd8-baaa-4905-b3cc-237c00fdcb73","Type":"ContainerDied","Data":"f5d97d89fcb917ea9e8b0eeeb8f2b0d20c087cef6217422c7040120e231a721f"} Nov 03 22:30:30.386230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.386203 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:30.468730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.468633 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-xks-ca\") pod \"818eafd8-baaa-4905-b3cc-237c00fdcb73\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " Nov 03 22:30:30.468730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.468718 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818eafd8-baaa-4905-b3cc-237c00fdcb73-kube-api-access-6hjn4\" (UniqueName: \"kubernetes.io/projected/818eafd8-baaa-4905-b3cc-237c00fdcb73-kube-api-access-6hjn4\") pod \"818eafd8-baaa-4905-b3cc-237c00fdcb73\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " Nov 03 22:30:30.468960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.468756 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-openshift-ca\") pod \"818eafd8-baaa-4905-b3cc-237c00fdcb73\" (UID: \"818eafd8-baaa-4905-b3cc-237c00fdcb73\") " Nov 03 22:30:30.469558 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.469515 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-openshift-ca" pod "818eafd8-baaa-4905-b3cc-237c00fdcb73" (UID: "818eafd8-baaa-4905-b3cc-237c00fdcb73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:30.471695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.471633 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-xks-ca" pod "818eafd8-baaa-4905-b3cc-237c00fdcb73" (UID: "818eafd8-baaa-4905-b3cc-237c00fdcb73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:30.471979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.471960 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818eafd8-baaa-4905-b3cc-237c00fdcb73-kube-api-access-6hjn4" pod "818eafd8-baaa-4905-b3cc-237c00fdcb73" (UID: "818eafd8-baaa-4905-b3cc-237c00fdcb73"). InnerVolumeSpecName "kube-api-access-6hjn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:30.570852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.570810 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:30.570852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.570850 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818eafd8-baaa-4905-b3cc-237c00fdcb73-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:30.570852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:30.570860 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hjn4\" (UniqueName: \"kubernetes.io/projected/818eafd8-baaa-4905-b3cc-237c00fdcb73-kube-api-access-6hjn4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:31.033106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:31.033065 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc"] Nov 03 22:30:31.038790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:31.038763 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbf6e63-644lc"] Nov 03 22:30:31.217214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:31.217184 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" Nov 03 22:30:31.217416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:31.217225 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q" event={"ID":"818eafd8-baaa-4905-b3cc-237c00fdcb73","Type":"ContainerDied","Data":"63b37bc1b5ca5a429cebee5e0ea4f865fc28538a9500913964cc25c20af4ecd4"} Nov 03 22:30:31.217416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:31.217281 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63b37bc1b5ca5a429cebee5e0ea4f865fc28538a9500913964cc25c20af4ecd4" Nov 03 22:30:31.741228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:31.740435 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f3f75d-be17-4f2a-ba55-1f70316410af" path="/var/lib/kubelet/pods/55f3f75d-be17-4f2a-ba55-1f70316410af/volumes" Nov 03 22:30:33.106833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:33.106801 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8"] Nov 03 22:30:33.115255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:33.115223 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ef02f9-6f6m8"] Nov 03 22:30:33.735483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:33.735451 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e55a11b-c674-49f4-9090-caedaa7dff18" path="/var/lib/kubelet/pods/0e55a11b-c674-49f4-9090-caedaa7dff18/volumes" Nov 03 22:30:41.062254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:41.062192 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6"] Nov 03 22:30:41.074597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:41.074535 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn"] Nov 03 22:30:41.085102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:41.085038 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eff0233e-45pc6"] Nov 03 22:30:41.098859 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:41.098826 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92682aab-47zwn"] Nov 03 22:30:41.790312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:41.778098 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980313da-60b3-45e8-935e-96cfd1f77812" path="/var/lib/kubelet/pods/980313da-60b3-45e8-935e-96cfd1f77812/volumes" Nov 03 22:30:41.791588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:41.791559 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36caeba-0f0a-4345-bbcb-96b6cfd57913" path="/var/lib/kubelet/pods/c36caeba-0f0a-4345-bbcb-96b6cfd57913/volumes" Nov 03 22:30:42.095456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.095357 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4"] Nov 03 22:30:42.153317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.145122 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d1afc4-dzrx4"] Nov 03 22:30:42.472418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.472373 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:42.500420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.500377 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:42.535427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.535375 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:42.637425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.637375 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:30:42.666456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.666410 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:42.730222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.730129 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:42.744778 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.744735 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:42.745351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.745325 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 22:30:42.749063 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:42.746248 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:43.071482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.071376 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:43.190457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.190376 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:43.280368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.279111 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:43.506879 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.506839 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:43.509262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.509131 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:30:43.556319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.537467 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 22:30:43.556319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.554446 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:43.582876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.557068 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg"] Nov 03 22:30:43.607314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.584229 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerName="cleanup" Nov 03 22:30:43.608865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.608836 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerName="cleanup" Nov 03 22:30:43.609023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.609012 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d446e624-f0c0-4901-8542-a332384a8b82" containerName="cleanup" Nov 03 22:30:43.630529 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.609101 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d446e624-f0c0-4901-8542-a332384a8b82" containerName="cleanup" Nov 03 22:30:43.630801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.630777 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerName="cleanup" Nov 03 22:30:43.630918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.630904 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerName="cleanup" Nov 03 22:30:43.631030 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.631018 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerName="cleanup" Nov 03 22:30:43.631119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.631108 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ee5b23-19dd-4f7c-83e7-470f31eb71f6" containerName="cleanup" Nov 03 22:30:43.631258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.631245 4903 cpu_manager.go:414] "containerMap: removing container" podUID="818eafd8-baaa-4905-b3cc-237c00fdcb73" containerName="revoke-keys" Nov 03 22:30:43.631377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.631363 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818eafd8-baaa-4905-b3cc-237c00fdcb73" containerName="revoke-keys" Nov 03 22:30:43.631502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.631489 4903 cpu_manager.go:414] "containerMap: removing container" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" containerName="cleanup" Nov 03 22:30:43.631587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.631575 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" containerName="cleanup" Nov 03 22:30:43.639713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.639685 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="818eafd8-baaa-4905-b3cc-237c00fdcb73" containerName="revoke-keys" Nov 03 22:30:43.639881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.639867 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d446e624-f0c0-4901-8542-a332384a8b82" containerName="cleanup" Nov 03 22:30:43.639983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.639972 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c583acd-e85b-4b5c-83d8-0b5e7774dbc9" containerName="cleanup" Nov 03 22:30:43.640076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.640066 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="06af2cbf-605f-41db-83be-08545aaa5b31" containerName="cleanup" Nov 03 22:30:43.650592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.650563 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b112bfd-7ef9-4e26-954c-3627c4a3e648" containerName="cleanup" Nov 03 22:30:43.688315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.655818 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:43.688315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.684056 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 03 22:30:43.688315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.684630 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:30:43.725321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.712919 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:30:43.940377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.726711 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:30:43.940377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.726765 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:30:43.940377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.726847 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:30:43.940377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.829477 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:30:43.940377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.829515 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 22:30:43.940377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.868431 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d05919ef-7nklg\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:43.940377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.928101 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcs7h\" (UniqueName: \"kubernetes.io/projected/5522297d-63fe-41b5-bc5a-fd983abe27fd-kube-api-access-bcs7h\") pod \"maas-api-revoke-keys-e2e-ait-d05919ef-7nklg\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:43.940377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.928143 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d05919ef-7nklg\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:43.940377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.905155 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:43.962342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.957815 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-qhqx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:30:43.962342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:43.957887 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:30:44.046418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.043913 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d05919ef-7nklg\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:44.059635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.051912 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcs7h\" (UniqueName: \"kubernetes.io/projected/5522297d-63fe-41b5-bc5a-fd983abe27fd-kube-api-access-bcs7h\") pod \"maas-api-revoke-keys-e2e-ait-d05919ef-7nklg\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:44.059635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.053546 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d05919ef-7nklg\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:44.059635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.056380 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d05919ef-7nklg\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:44.107666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.095030 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcs7h\" (UniqueName: \"kubernetes.io/projected/5522297d-63fe-41b5-bc5a-fd983abe27fd-kube-api-access-bcs7h\") pod \"maas-api-revoke-keys-e2e-ait-d05919ef-7nklg\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:44.107666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.105083 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8719195-685c-4781-908b-58a9d3292b8c" path="/var/lib/kubelet/pods/f8719195-685c-4781-908b-58a9d3292b8c/volumes" Nov 03 22:30:44.107666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.105541 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg"] Nov 03 22:30:44.107666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.105634 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:30:44.107666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.105673 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:30:44.107666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.105702 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 22:30:44.111978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.111944 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 22:30:44.116653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.112843 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d05919ef-7nklg\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:44.179653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.179623 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 22:30:44.338165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.338082 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:44.711056 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:30:44.711000 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5522297d_63fe_41b5_bc5a_fd983abe27fd.slice/crio-3422e1a1deea62bc58e0f964824286abea67ff403d0cf93962aa76ad8345d604 WatchSource:0}: Error finding container 3422e1a1deea62bc58e0f964824286abea67ff403d0cf93962aa76ad8345d604: Status 404 returned error can't find the container with id 3422e1a1deea62bc58e0f964824286abea67ff403d0cf93962aa76ad8345d604 Nov 03 22:30:44.712421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:44.712398 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg"] Nov 03 22:30:45.530268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:45.530210 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" event={"ID":"5522297d-63fe-41b5-bc5a-fd983abe27fd","Type":"ContainerStarted","Data":"a7d08bc9a5875f293fe6f6a2f84c32645b3da9795a50e18544d1e8d5ac9377f1"} Nov 03 22:30:45.533500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:45.533467 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" event={"ID":"5522297d-63fe-41b5-bc5a-fd983abe27fd","Type":"ContainerStarted","Data":"3422e1a1deea62bc58e0f964824286abea67ff403d0cf93962aa76ad8345d604"} Nov 03 22:30:45.573523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:45.573460 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" podStartSLOduration=2.5734382289999997 podStartE2EDuration="2.573438229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:30:45.572512443 +0000 UTC m=+1071.976453419" watchObservedRunningTime="2026-11-03 22:30:45.573438229 +0000 UTC m=+1071.977379205" Nov 03 22:30:46.543072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:46.542984 4903 generic.go:356] "Generic (PLEG): container finished" podID="5522297d-63fe-41b5-bc5a-fd983abe27fd" containerID="a7d08bc9a5875f293fe6f6a2f84c32645b3da9795a50e18544d1e8d5ac9377f1" exitCode=0 Nov 03 22:30:46.543072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:46.543060 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" event={"ID":"5522297d-63fe-41b5-bc5a-fd983abe27fd","Type":"ContainerDied","Data":"a7d08bc9a5875f293fe6f6a2f84c32645b3da9795a50e18544d1e8d5ac9377f1"} Nov 03 22:30:47.731968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.731942 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:47.820187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.820104 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5522297d-63fe-41b5-bc5a-fd983abe27fd-kube-api-access-bcs7h\" (UniqueName: \"kubernetes.io/projected/5522297d-63fe-41b5-bc5a-fd983abe27fd-kube-api-access-bcs7h\") pod \"5522297d-63fe-41b5-bc5a-fd983abe27fd\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " Nov 03 22:30:47.820415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.820392 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-xks-ca\") pod \"5522297d-63fe-41b5-bc5a-fd983abe27fd\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " Nov 03 22:30:47.820508 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.820479 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-openshift-ca\") pod \"5522297d-63fe-41b5-bc5a-fd983abe27fd\" (UID: \"5522297d-63fe-41b5-bc5a-fd983abe27fd\") " Nov 03 22:30:47.820886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.820852 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-openshift-ca" pod "5522297d-63fe-41b5-bc5a-fd983abe27fd" (UID: "5522297d-63fe-41b5-bc5a-fd983abe27fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:47.821445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.821421 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:47.822647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.822622 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5522297d-63fe-41b5-bc5a-fd983abe27fd-kube-api-access-bcs7h" pod "5522297d-63fe-41b5-bc5a-fd983abe27fd" (UID: "5522297d-63fe-41b5-bc5a-fd983abe27fd"). InnerVolumeSpecName "kube-api-access-bcs7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:47.822733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.822633 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-xks-ca" pod "5522297d-63fe-41b5-bc5a-fd983abe27fd" (UID: "5522297d-63fe-41b5-bc5a-fd983abe27fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:47.923085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.923041 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcs7h\" (UniqueName: \"kubernetes.io/projected/5522297d-63fe-41b5-bc5a-fd983abe27fd-kube-api-access-bcs7h\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:47.923085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:47.923081 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5522297d-63fe-41b5-bc5a-fd983abe27fd-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:48.385005 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:30:48.384972 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:30:48.562588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:48.562558 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" Nov 03 22:30:48.562759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:48.562555 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg" event={"ID":"5522297d-63fe-41b5-bc5a-fd983abe27fd","Type":"ContainerDied","Data":"3422e1a1deea62bc58e0f964824286abea67ff403d0cf93962aa76ad8345d604"} Nov 03 22:30:48.562759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:48.562705 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3422e1a1deea62bc58e0f964824286abea67ff403d0cf93962aa76ad8345d604" Nov 03 22:30:51.032804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:51.032769 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6"] Nov 03 22:30:51.041850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:51.041817 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f20ab1-l7vg6"] Nov 03 22:30:51.745319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:51.744813 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a73ea1e-ec66-46c5-a37b-aef249c3bc51" path="/var/lib/kubelet/pods/1a73ea1e-ec66-46c5-a37b-aef249c3bc51/volumes" Nov 03 22:30:52.033968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.032761 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5"] Nov 03 22:30:52.042810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.042777 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e8d15ed-7lrr5"] Nov 03 22:30:52.827502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.827468 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595"] Nov 03 22:30:52.828190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.828176 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5522297d-63fe-41b5-bc5a-fd983abe27fd" containerName="revoke-keys" Nov 03 22:30:52.828237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.828192 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5522297d-63fe-41b5-bc5a-fd983abe27fd" containerName="revoke-keys" Nov 03 22:30:52.828346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.828333 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5522297d-63fe-41b5-bc5a-fd983abe27fd" containerName="revoke-keys" Nov 03 22:30:52.832282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.832267 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:52.848430 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.848402 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595"] Nov 03 22:30:52.854337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.852791 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:30:52.854337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.852823 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:30:52.854337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.852857 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:30:52.854337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.852892 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:30:52.899976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.899941 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9987l\" (UniqueName: \"kubernetes.io/projected/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-kube-api-access-9987l\") pod \"maas-api-revoke-keys-e2e-derive-cbe6e297-cg595\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:52.900136 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.900055 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-cbe6e297-cg595\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:52.900180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:52.900153 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-cbe6e297-cg595\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:53.001437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.001400 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9987l\" (UniqueName: \"kubernetes.io/projected/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-kube-api-access-9987l\") pod \"maas-api-revoke-keys-e2e-derive-cbe6e297-cg595\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:53.001624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.001477 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-cbe6e297-cg595\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:53.001624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.001582 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-cbe6e297-cg595\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:53.002265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.002239 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-cbe6e297-cg595\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:53.003903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.003878 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-cbe6e297-cg595\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:53.023015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.022983 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9987l\" (UniqueName: \"kubernetes.io/projected/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-kube-api-access-9987l\") pod \"maas-api-revoke-keys-e2e-derive-cbe6e297-cg595\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:53.147111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.147026 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:53.454010 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:30:53.453978 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7a1b4b_8c01_4bbb_83bd_c2c92d42c9ef.slice/crio-7763ce4e2b71443b6236b6b8a38045a946f194e3d6dbf8a6881913c11205d78b WatchSource:0}: Error finding container 7763ce4e2b71443b6236b6b8a38045a946f194e3d6dbf8a6881913c11205d78b: Status 404 returned error can't find the container with id 7763ce4e2b71443b6236b6b8a38045a946f194e3d6dbf8a6881913c11205d78b Nov 03 22:30:53.456664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.456640 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595"] Nov 03 22:30:53.615834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.615798 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" event={"ID":"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef","Type":"ContainerStarted","Data":"7763ce4e2b71443b6236b6b8a38045a946f194e3d6dbf8a6881913c11205d78b"} Nov 03 22:30:53.739957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:53.739871 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c6c00e-3801-48bb-853f-72ec8ef5d831" path="/var/lib/kubelet/pods/47c6c00e-3801-48bb-853f-72ec8ef5d831/volumes" Nov 03 22:30:54.645525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:54.645433 4903 generic.go:356] "Generic (PLEG): container finished" podID="5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef" containerID="53a564f5c975d16b113032ee57634f3ea0e25363d78397e96259648a113cf32a" exitCode=0 Nov 03 22:30:54.645897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:54.645527 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" event={"ID":"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef","Type":"ContainerDied","Data":"53a564f5c975d16b113032ee57634f3ea0e25363d78397e96259648a113cf32a"} Nov 03 22:30:55.462586 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:55.462556 4903 scope.go:122] "RemoveContainer" containerID="3df1d1c301d011e8ef538040c32c01c9bd60ad5081558c91c219981c6b09de28" Nov 03 22:30:55.841920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:55.841891 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:55.959515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:55.959479 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-xks-ca\") pod \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " Nov 03 22:30:55.959721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:55.959683 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-openshift-ca\") pod \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " Nov 03 22:30:55.959794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:55.959770 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-kube-api-access-9987l\" (UniqueName: \"kubernetes.io/projected/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-kube-api-access-9987l\") pod \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\" (UID: \"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef\") " Nov 03 22:30:55.960097 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:55.960066 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-openshift-ca" pod "5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef" (UID: "5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:30:55.960407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:55.960383 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:55.961872 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:55.961837 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-xks-ca" pod "5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef" (UID: "5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:30:55.961976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:55.961937 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-kube-api-access-9987l" pod "5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef" (UID: "5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef"). InnerVolumeSpecName "kube-api-access-9987l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:30:56.062398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:56.062351 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9987l\" (UniqueName: \"kubernetes.io/projected/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-kube-api-access-9987l\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:56.062398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:56.062389 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:30:56.665907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:56.665877 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" Nov 03 22:30:56.666096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:56.665914 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595" event={"ID":"5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef","Type":"ContainerDied","Data":"7763ce4e2b71443b6236b6b8a38045a946f194e3d6dbf8a6881913c11205d78b"} Nov 03 22:30:56.666096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:56.665968 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7763ce4e2b71443b6236b6b8a38045a946f194e3d6dbf8a6881913c11205d78b" Nov 03 22:30:58.851361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:58.841728 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg"] Nov 03 22:30:58.854760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:58.854727 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d05919ef-7nklg"] Nov 03 22:30:59.736847 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:30:59.736816 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5522297d-63fe-41b5-bc5a-fd983abe27fd" path="/var/lib/kubelet/pods/5522297d-63fe-41b5-bc5a-fd983abe27fd/volumes" Nov 03 22:31:07.482482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:07.482444 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595"] Nov 03 22:31:07.491311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:07.491272 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cbe6e297-cg595"] Nov 03 22:31:07.753931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:07.753848 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef" path="/var/lib/kubelet/pods/5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef/volumes" Nov 03 22:31:09.135029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.134997 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g"] Nov 03 22:31:09.135777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.135759 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef" containerName="revoke-keys" Nov 03 22:31:09.135777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.135777 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef" containerName="revoke-keys" Nov 03 22:31:09.135898 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.135889 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7a1b4b-8c01-4bbb-83bd-c2c92d42c9ef" containerName="revoke-keys" Nov 03 22:31:09.141689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.141669 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:09.170580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.170552 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-281732\"" Nov 03 22:31:09.171254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.171180 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-281732-dockercfg-v46gf\"" Nov 03 22:31:09.207234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.191507 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g"] Nov 03 22:31:09.209464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.209431 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4pj\" (UniqueName: \"kubernetes.io/projected/42016bc2-bc6d-43af-a7df-658b02f4a602-kube-api-access-9h4pj\") pod \"payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g\" (UID: \"42016bc2-bc6d-43af-a7df-658b02f4a602\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:09.209909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.209886 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42016bc2-bc6d-43af-a7df-658b02f4a602-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g\" (UID: \"42016bc2-bc6d-43af-a7df-658b02f4a602\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:09.329824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.324825 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42016bc2-bc6d-43af-a7df-658b02f4a602-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g\" (UID: \"42016bc2-bc6d-43af-a7df-658b02f4a602\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:09.330445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.330406 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4pj\" (UniqueName: \"kubernetes.io/projected/42016bc2-bc6d-43af-a7df-658b02f4a602-kube-api-access-9h4pj\") pod \"payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g\" (UID: \"42016bc2-bc6d-43af-a7df-658b02f4a602\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:09.378283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.378247 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42016bc2-bc6d-43af-a7df-658b02f4a602-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g\" (UID: \"42016bc2-bc6d-43af-a7df-658b02f4a602\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:09.400428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.400405 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4pj\" (UniqueName: \"kubernetes.io/projected/42016bc2-bc6d-43af-a7df-658b02f4a602-kube-api-access-9h4pj\") pod \"payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g\" (UID: \"42016bc2-bc6d-43af-a7df-658b02f4a602\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:09.492151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.492112 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:09.821100 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:31:09.821064 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42016bc2_bc6d_43af_a7df_658b02f4a602.slice/crio-fc007ba4e55fa90c8764014b84408f25f73a571b904fac4f5ff8cf52b09d0f67 WatchSource:0}: Error finding container fc007ba4e55fa90c8764014b84408f25f73a571b904fac4f5ff8cf52b09d0f67: Status 404 returned error can't find the container with id fc007ba4e55fa90c8764014b84408f25f73a571b904fac4f5ff8cf52b09d0f67 Nov 03 22:31:09.822341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:09.822312 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g"] Nov 03 22:31:10.823940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:10.823908 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" event={"ID":"42016bc2-bc6d-43af-a7df-658b02f4a602","Type":"ContainerStarted","Data":"0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b"} Nov 03 22:31:10.824385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:10.823960 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" event={"ID":"42016bc2-bc6d-43af-a7df-658b02f4a602","Type":"ContainerStarted","Data":"fc007ba4e55fa90c8764014b84408f25f73a571b904fac4f5ff8cf52b09d0f67"} Nov 03 22:31:10.824385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:10.824169 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:10.864135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:10.864083 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" podStartSLOduration=1.864067497 podStartE2EDuration="1.864067497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:10.861172365 +0000 UTC m=+1097.265113341" watchObservedRunningTime="2026-11-03 22:31:10.864067497 +0000 UTC m=+1097.268008472" Nov 03 22:31:18.385502 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:31:18.385463 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:31:18.622415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:18.622378 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:31:21.894319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:21.884470 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 22:31:48.446621 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:31:48.446578 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:31:49.110725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.110692 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5"] Nov 03 22:31:49.115890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.115867 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:31:49.154830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.154798 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-04988c\"" Nov 03 22:31:49.155221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.155191 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-04988c-dockercfg-249np\"" Nov 03 22:31:49.161644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.161615 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5"] Nov 03 22:31:49.303327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.299351 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7"] Nov 03 22:31:49.305678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.305649 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:31:49.335759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.335728 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7"] Nov 03 22:31:49.338457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.338413 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncxqt\" (UniqueName: \"kubernetes.io/projected/990c7438-dadc-4591-88ca-ab96519f2dec-kube-api-access-ncxqt\") pod \"payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5\" (UID: \"990c7438-dadc-4591-88ca-ab96519f2dec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:31:49.338777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.338756 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990c7438-dadc-4591-88ca-ab96519f2dec-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5\" (UID: \"990c7438-dadc-4591-88ca-ab96519f2dec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:31:49.440837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.440801 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncxqt\" (UniqueName: \"kubernetes.io/projected/990c7438-dadc-4591-88ca-ab96519f2dec-kube-api-access-ncxqt\") pod \"payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5\" (UID: \"990c7438-dadc-4591-88ca-ab96519f2dec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:31:49.441119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.440874 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1142a1b-aebb-4d30-af4b-1b5d716459ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7\" (UID: \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:31:49.441119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.440954 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990c7438-dadc-4591-88ca-ab96519f2dec-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5\" (UID: \"990c7438-dadc-4591-88ca-ab96519f2dec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:31:49.441119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.440995 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntj28\" (UniqueName: \"kubernetes.io/projected/b1142a1b-aebb-4d30-af4b-1b5d716459ec-kube-api-access-ntj28\") pod \"payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7\" (UID: \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:31:49.442149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.442121 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990c7438-dadc-4591-88ca-ab96519f2dec-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5\" (UID: \"990c7438-dadc-4591-88ca-ab96519f2dec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:31:49.463398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.463362 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncxqt\" (UniqueName: \"kubernetes.io/projected/990c7438-dadc-4591-88ca-ab96519f2dec-kube-api-access-ncxqt\") pod \"payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5\" (UID: \"990c7438-dadc-4591-88ca-ab96519f2dec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:31:49.542581 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.542547 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntj28\" (UniqueName: \"kubernetes.io/projected/b1142a1b-aebb-4d30-af4b-1b5d716459ec-kube-api-access-ntj28\") pod \"payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7\" (UID: \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:31:49.542841 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.542824 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1142a1b-aebb-4d30-af4b-1b5d716459ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7\" (UID: \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:31:49.543568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.543546 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1142a1b-aebb-4d30-af4b-1b5d716459ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7\" (UID: \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:31:49.564662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.564636 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntj28\" (UniqueName: \"kubernetes.io/projected/b1142a1b-aebb-4d30-af4b-1b5d716459ec-kube-api-access-ntj28\") pod \"payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7\" (UID: \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:31:49.706506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.706468 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:31:49.733914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:49.733885 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:31:50.033473 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:31:50.033436 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1142a1b_aebb_4d30_af4b_1b5d716459ec.slice/crio-8bcbd06ce609efeaf2b9f1b2bc7e13da28994eb6f696b773b540f6845b5d381e WatchSource:0}: Error finding container 8bcbd06ce609efeaf2b9f1b2bc7e13da28994eb6f696b773b540f6845b5d381e: Status 404 returned error can't find the container with id 8bcbd06ce609efeaf2b9f1b2bc7e13da28994eb6f696b773b540f6845b5d381e Nov 03 22:31:50.033740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.033722 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7"] Nov 03 22:31:50.076100 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:31:50.076056 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990c7438_dadc_4591_88ca_ab96519f2dec.slice/crio-01cf7d124c3c1162d23557b3dd576e8fb5edb7534c0d2eed2f1dfb0f38f49359 WatchSource:0}: Error finding container 01cf7d124c3c1162d23557b3dd576e8fb5edb7534c0d2eed2f1dfb0f38f49359: Status 404 returned error can't find the container with id 01cf7d124c3c1162d23557b3dd576e8fb5edb7534c0d2eed2f1dfb0f38f49359 Nov 03 22:31:50.077098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.077073 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5"] Nov 03 22:31:50.411097 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.411067 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" event={"ID":"990c7438-dadc-4591-88ca-ab96519f2dec","Type":"ContainerStarted","Data":"7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6"} Nov 03 22:31:50.411289 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.411122 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" event={"ID":"990c7438-dadc-4591-88ca-ab96519f2dec","Type":"ContainerStarted","Data":"01cf7d124c3c1162d23557b3dd576e8fb5edb7534c0d2eed2f1dfb0f38f49359"} Nov 03 22:31:50.411289 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.411168 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:31:50.413110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.413083 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" event={"ID":"b1142a1b-aebb-4d30-af4b-1b5d716459ec","Type":"ContainerStarted","Data":"562db4f092ef27ce6179113125e434689dfd83a6c575a524ed33831e7b53bf26"} Nov 03 22:31:50.413224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.413129 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" event={"ID":"b1142a1b-aebb-4d30-af4b-1b5d716459ec","Type":"ContainerStarted","Data":"8bcbd06ce609efeaf2b9f1b2bc7e13da28994eb6f696b773b540f6845b5d381e"} Nov 03 22:31:50.413224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.413159 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:31:50.453589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.453539 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" podStartSLOduration=1.45352447 podStartE2EDuration="1.45352447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:50.447118649 +0000 UTC m=+1136.851059629" watchObservedRunningTime="2026-11-03 22:31:50.45352447 +0000 UTC m=+1136.857465462" Nov 03 22:31:50.485825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:50.485778 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" podStartSLOduration=1.48575228 podStartE2EDuration="1.48575228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:31:50.482806445 +0000 UTC m=+1136.886747420" watchObservedRunningTime="2026-11-03 22:31:50.48575228 +0000 UTC m=+1136.889693255" Nov 03 22:31:51.033719 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:51.033683 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l"] Nov 03 22:31:51.041005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:51.040971 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd88531e-fw92l"] Nov 03 22:31:51.737084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:51.737043 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322502ba-fa0c-401b-91f8-940487528ef1" path="/var/lib/kubelet/pods/322502ba-fa0c-401b-91f8-940487528ef1/volumes" Nov 03 22:31:55.565908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.565878 4903 scope.go:122] "RemoveContainer" containerID="6112ce8f6c744ab8ba0567a49de34b2914cd548602e1755dba9174c6c2a17014" Nov 03 22:31:55.578155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.578130 4903 scope.go:122] "RemoveContainer" containerID="4e7c71364bf28068d687435c09b450e281bc497110f60eae48f4cabaf1c24389" Nov 03 22:31:55.592879 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.592853 4903 scope.go:122] "RemoveContainer" containerID="c27b41d7aabaf7f6648391e7ba35373eedf15bcba6d1544b16cc94185b927060" Nov 03 22:31:55.605681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.605653 4903 scope.go:122] "RemoveContainer" containerID="1d915e4ff8b5e29824984cf7ab7fa4a7b498265472d8a1891eb0951a814e956b" Nov 03 22:31:55.624894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.624870 4903 scope.go:122] "RemoveContainer" containerID="c849268fbadbb9fe0d461370746f2e4f17073b73b9d1a13a787d367d72250a65" Nov 03 22:31:55.638574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.638551 4903 scope.go:122] "RemoveContainer" containerID="597c9f0c698880cca3cf8bb6ac22a6c416e354d8cf06d91bd7fe72816ce1d38b" Nov 03 22:31:55.649593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.649573 4903 scope.go:122] "RemoveContainer" containerID="5ed6888275a4df08bd5bb0924adc6f0a5ad255e97177fa1c1d90b8c153534952" Nov 03 22:31:55.661061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.661041 4903 scope.go:122] "RemoveContainer" containerID="5dd3f77f6c8d6d317ae6ee2a9964947b8909bfa06736ae089616b29ff34fb821" Nov 03 22:31:55.672390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.672361 4903 scope.go:122] "RemoveContainer" containerID="41fb44baa609bf5b9c05b86c128016568b0bd6055373d429ec0dcf9304d253fa" Nov 03 22:31:55.683614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.683590 4903 scope.go:122] "RemoveContainer" containerID="b4602f14160cf569e2886d91b5d80b9eab7d1261936b027bd6b065fb2ea52857" Nov 03 22:31:55.702931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.702906 4903 scope.go:122] "RemoveContainer" containerID="079caadab189907d9155f482a04dd7928cf0b76ee2fb1256a541f7a5fa7f11e7" Nov 03 22:31:55.715665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.715637 4903 scope.go:122] "RemoveContainer" containerID="78f18bcf4fd51d6fd680b322c36ce02d5aefd349e62d717b6c8e61080423e2c0" Nov 03 22:31:55.730419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.730395 4903 scope.go:122] "RemoveContainer" containerID="9ab9eaa7883482b4bf46a9af5a88aeee7e8d1994cadc2e51e2fe68e923edca7b" Nov 03 22:31:55.744965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:31:55.744921 4903 scope.go:122] "RemoveContainer" containerID="cb343e68b515a1681fd9d136fdee97c6393b6b64347b06f4c8b576ea78f5fe6a" Nov 03 22:32:01.439336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.439267 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 22:32:01.439787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.439412 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 22:32:01.752922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.752845 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv"] Nov 03 22:32:01.757482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.757459 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:01.768503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.768473 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2q6\" (UniqueName: \"kubernetes.io/projected/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-kube-api-access-4v2q6\") pod \"payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv\" (UID: \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:01.768684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.768661 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv\" (UID: \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:01.770704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.770682 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-a940d5\"" Nov 03 22:32:01.771393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.771369 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-a940d5-dockercfg-p9z6c\"" Nov 03 22:32:01.778828 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.778801 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv"] Nov 03 22:32:01.817465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.817427 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz"] Nov 03 22:32:01.879483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.879435 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2q6\" (UniqueName: \"kubernetes.io/projected/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-kube-api-access-4v2q6\") pod \"payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv\" (UID: \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:01.891230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.891192 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv\" (UID: \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:01.898500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.898065 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz"] Nov 03 22:32:01.898500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.898225 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:01.907317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.907269 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv\" (UID: \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:01.978315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.976083 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9"] Nov 03 22:32:01.995640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.995577 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5l9q\" (UniqueName: \"kubernetes.io/projected/1c106a89-020e-4116-9077-19f6304b4685-kube-api-access-g5l9q\") pod \"payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz\" (UID: \"1c106a89-020e-4116-9077-19f6304b4685\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:01.996155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.996107 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c106a89-020e-4116-9077-19f6304b4685-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz\" (UID: \"1c106a89-020e-4116-9077-19f6304b4685\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:02.003336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:01.999389 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2q6\" (UniqueName: \"kubernetes.io/projected/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-kube-api-access-4v2q6\") pod \"payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv\" (UID: \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:02.003336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.002493 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9"] Nov 03 22:32:02.003336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.002604 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.016245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.016214 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:32:02.016442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.016383 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:32:02.073512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.073468 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:02.093882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.093674 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:32:02.093882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.093775 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:32:02.097942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.097908 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a7419de-nzkr9\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.098093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.098021 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a7419de-nzkr9\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.098204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.098189 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58xz\" (UniqueName: \"kubernetes.io/projected/cdd935e3-a795-44e5-a570-39a00ef9bbaa-kube-api-access-l58xz\") pod \"maas-api-revoke-sub-6a7419de-nzkr9\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.098272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.098258 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5l9q\" (UniqueName: \"kubernetes.io/projected/1c106a89-020e-4116-9077-19f6304b4685-kube-api-access-g5l9q\") pod \"payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz\" (UID: \"1c106a89-020e-4116-9077-19f6304b4685\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:02.098362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.098341 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c106a89-020e-4116-9077-19f6304b4685-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz\" (UID: \"1c106a89-020e-4116-9077-19f6304b4685\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:02.099189 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.099166 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c106a89-020e-4116-9077-19f6304b4685-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz\" (UID: \"1c106a89-020e-4116-9077-19f6304b4685\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:02.121783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.121740 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5l9q\" (UniqueName: \"kubernetes.io/projected/1c106a89-020e-4116-9077-19f6304b4685-kube-api-access-g5l9q\") pod \"payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz\" (UID: \"1c106a89-020e-4116-9077-19f6304b4685\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:02.213854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.206918 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58xz\" (UniqueName: \"kubernetes.io/projected/cdd935e3-a795-44e5-a570-39a00ef9bbaa-kube-api-access-l58xz\") pod \"maas-api-revoke-sub-6a7419de-nzkr9\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.213854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.207105 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a7419de-nzkr9\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.213854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.207222 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a7419de-nzkr9\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.213854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.208150 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a7419de-nzkr9\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.218028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.217942 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a7419de-nzkr9\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.232175 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.232136 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58xz\" (UniqueName: \"kubernetes.io/projected/cdd935e3-a795-44e5-a570-39a00ef9bbaa-kube-api-access-l58xz\") pod \"maas-api-revoke-sub-6a7419de-nzkr9\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.283534 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.283439 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:02.331440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.331405 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:02.423566 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:32:02.423524 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3f5424_1ea1_4178_88e3_64b8f00bf1fd.slice/crio-d01c7ac47a7c84dc5b83032fd987506e536d326201f507559772b5dc16b41024 WatchSource:0}: Error finding container d01c7ac47a7c84dc5b83032fd987506e536d326201f507559772b5dc16b41024: Status 404 returned error can't find the container with id d01c7ac47a7c84dc5b83032fd987506e536d326201f507559772b5dc16b41024 Nov 03 22:32:02.425533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.425505 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv"] Nov 03 22:32:02.595799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.595716 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" event={"ID":"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd","Type":"ContainerStarted","Data":"a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a"} Nov 03 22:32:02.595799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.595776 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" event={"ID":"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd","Type":"ContainerStarted","Data":"d01c7ac47a7c84dc5b83032fd987506e536d326201f507559772b5dc16b41024"} Nov 03 22:32:02.596222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.595803 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:02.606449 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:32:02.606423 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c106a89_020e_4116_9077_19f6304b4685.slice/crio-35c6e1f39f800731cac0429f0ec0cc5ff3bb0f68a3a2e01a9c9e5c5f842b551a WatchSource:0}: Error finding container 35c6e1f39f800731cac0429f0ec0cc5ff3bb0f68a3a2e01a9c9e5c5f842b551a: Status 404 returned error can't find the container with id 35c6e1f39f800731cac0429f0ec0cc5ff3bb0f68a3a2e01a9c9e5c5f842b551a Nov 03 22:32:02.607315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.607280 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz"] Nov 03 22:32:02.636622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.636566 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" podStartSLOduration=1.636548133 podStartE2EDuration="1.636548133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:02.634755715 +0000 UTC m=+1149.038696690" watchObservedRunningTime="2026-11-03 22:32:02.636548133 +0000 UTC m=+1149.040489108" Nov 03 22:32:02.696590 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:32:02.696559 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd935e3_a795_44e5_a570_39a00ef9bbaa.slice/crio-70885247a757ad406676c5ba97456647fdb03aab77c7184b4de7d6cafe78c395 WatchSource:0}: Error finding container 70885247a757ad406676c5ba97456647fdb03aab77c7184b4de7d6cafe78c395: Status 404 returned error can't find the container with id 70885247a757ad406676c5ba97456647fdb03aab77c7184b4de7d6cafe78c395 Nov 03 22:32:02.698539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:02.698517 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9"] Nov 03 22:32:03.611856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:03.611817 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" event={"ID":"cdd935e3-a795-44e5-a570-39a00ef9bbaa","Type":"ContainerStarted","Data":"b4cc6679772b80a630b979ea378b76709e2fadf6dd69075d0cf87fb2bc3c24b8"} Nov 03 22:32:03.612373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:03.611881 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" event={"ID":"cdd935e3-a795-44e5-a570-39a00ef9bbaa","Type":"ContainerStarted","Data":"70885247a757ad406676c5ba97456647fdb03aab77c7184b4de7d6cafe78c395"} Nov 03 22:32:03.613729 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:03.613700 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" event={"ID":"1c106a89-020e-4116-9077-19f6304b4685","Type":"ContainerStarted","Data":"77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe"} Nov 03 22:32:03.613855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:03.613843 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:03.613923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:03.613877 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" event={"ID":"1c106a89-020e-4116-9077-19f6304b4685","Type":"ContainerStarted","Data":"35c6e1f39f800731cac0429f0ec0cc5ff3bb0f68a3a2e01a9c9e5c5f842b551a"} Nov 03 22:32:03.662926 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:03.662880 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" podStartSLOduration=2.662865785 podStartE2EDuration="2.662865785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:03.648130211 +0000 UTC m=+1150.052071185" watchObservedRunningTime="2026-11-03 22:32:03.662865785 +0000 UTC m=+1150.066806782" Nov 03 22:32:03.689004 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:03.688947 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" podStartSLOduration=2.688933076 podStartE2EDuration="2.688933076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:03.685041787 +0000 UTC m=+1150.088982763" watchObservedRunningTime="2026-11-03 22:32:03.688933076 +0000 UTC m=+1150.092874051" Nov 03 22:32:04.034323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.034278 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f"] Nov 03 22:32:04.038885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.038865 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.058368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.058338 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f"] Nov 03 22:32:04.143176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.143119 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw27b\" (UniqueName: \"kubernetes.io/projected/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-kube-api-access-sw27b\") pod \"maas-api-revoke-keys-e2e-mt-53879c92-5vd6f\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.143432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.143337 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-53879c92-5vd6f\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.143533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.143514 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-53879c92-5vd6f\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.245143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.245104 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw27b\" (UniqueName: \"kubernetes.io/projected/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-kube-api-access-sw27b\") pod \"maas-api-revoke-keys-e2e-mt-53879c92-5vd6f\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.245350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.245224 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-53879c92-5vd6f\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.245415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.245357 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-53879c92-5vd6f\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.246007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.245980 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-53879c92-5vd6f\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.247653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.247624 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-53879c92-5vd6f\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.284221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.284190 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw27b\" (UniqueName: \"kubernetes.io/projected/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-kube-api-access-sw27b\") pod \"maas-api-revoke-keys-e2e-mt-53879c92-5vd6f\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.354834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.354744 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:04.625154 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.625058 4903 generic.go:356] "Generic (PLEG): container finished" podID="cdd935e3-a795-44e5-a570-39a00ef9bbaa" containerID="b4cc6679772b80a630b979ea378b76709e2fadf6dd69075d0cf87fb2bc3c24b8" exitCode=0 Nov 03 22:32:04.625663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.625158 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" event={"ID":"cdd935e3-a795-44e5-a570-39a00ef9bbaa","Type":"ContainerDied","Data":"b4cc6679772b80a630b979ea378b76709e2fadf6dd69075d0cf87fb2bc3c24b8"} Nov 03 22:32:04.671435 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:32:04.671396 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03a00fe_cc7e_4d1b_84d7_b45ed81901df.slice/crio-a316e69fb042ce81c8f84ae23c53a944fa08b7a22ee83d5e53f2913439d86a22 WatchSource:0}: Error finding container a316e69fb042ce81c8f84ae23c53a944fa08b7a22ee83d5e53f2913439d86a22: Status 404 returned error can't find the container with id a316e69fb042ce81c8f84ae23c53a944fa08b7a22ee83d5e53f2913439d86a22 Nov 03 22:32:04.677102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:04.677076 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f"] Nov 03 22:32:05.637273 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.637235 4903 generic.go:356] "Generic (PLEG): container finished" podID="c03a00fe-cc7e-4d1b-84d7-b45ed81901df" containerID="b8ccb680088ecb097ab62b9e2090cfc8aa543b680478c8ff3f00e67c23dd6817" exitCode=0 Nov 03 22:32:05.637683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.637327 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" event={"ID":"c03a00fe-cc7e-4d1b-84d7-b45ed81901df","Type":"ContainerDied","Data":"b8ccb680088ecb097ab62b9e2090cfc8aa543b680478c8ff3f00e67c23dd6817"} Nov 03 22:32:05.637683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.637391 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" event={"ID":"c03a00fe-cc7e-4d1b-84d7-b45ed81901df","Type":"ContainerStarted","Data":"a316e69fb042ce81c8f84ae23c53a944fa08b7a22ee83d5e53f2913439d86a22"} Nov 03 22:32:05.816445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.816419 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:05.873854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.873819 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-xks-ca\") pod \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " Nov 03 22:32:05.874039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.873906 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd935e3-a795-44e5-a570-39a00ef9bbaa-kube-api-access-l58xz\" (UniqueName: \"kubernetes.io/projected/cdd935e3-a795-44e5-a570-39a00ef9bbaa-kube-api-access-l58xz\") pod \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " Nov 03 22:32:05.874105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.874049 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-openshift-ca\") pod \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\" (UID: \"cdd935e3-a795-44e5-a570-39a00ef9bbaa\") " Nov 03 22:32:05.874779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.874748 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-openshift-ca" pod "cdd935e3-a795-44e5-a570-39a00ef9bbaa" (UID: "cdd935e3-a795-44e5-a570-39a00ef9bbaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:05.875006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.874982 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:32:05.877179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.877153 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd935e3-a795-44e5-a570-39a00ef9bbaa-kube-api-access-l58xz" pod "cdd935e3-a795-44e5-a570-39a00ef9bbaa" (UID: "cdd935e3-a795-44e5-a570-39a00ef9bbaa"). InnerVolumeSpecName "kube-api-access-l58xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:05.878268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.878247 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-xks-ca" pod "cdd935e3-a795-44e5-a570-39a00ef9bbaa" (UID: "cdd935e3-a795-44e5-a570-39a00ef9bbaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:05.976724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.976689 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd935e3-a795-44e5-a570-39a00ef9bbaa-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:32:05.976724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:05.976722 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l58xz\" (UniqueName: \"kubernetes.io/projected/cdd935e3-a795-44e5-a570-39a00ef9bbaa-kube-api-access-l58xz\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:32:06.647176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.647147 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" Nov 03 22:32:06.647176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.647170 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9" event={"ID":"cdd935e3-a795-44e5-a570-39a00ef9bbaa","Type":"ContainerDied","Data":"70885247a757ad406676c5ba97456647fdb03aab77c7184b4de7d6cafe78c395"} Nov 03 22:32:06.647672 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.647217 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70885247a757ad406676c5ba97456647fdb03aab77c7184b4de7d6cafe78c395" Nov 03 22:32:06.876644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.876616 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:06.899082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.899000 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-openshift-ca\") pod \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " Nov 03 22:32:06.899244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.899228 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-kube-api-access-sw27b\" (UniqueName: \"kubernetes.io/projected/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-kube-api-access-sw27b\") pod \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " Nov 03 22:32:06.899392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.899377 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-xks-ca\") pod \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\" (UID: \"c03a00fe-cc7e-4d1b-84d7-b45ed81901df\") " Nov 03 22:32:06.899476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.899385 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-openshift-ca" pod "c03a00fe-cc7e-4d1b-84d7-b45ed81901df" (UID: "c03a00fe-cc7e-4d1b-84d7-b45ed81901df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:06.900030 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.900007 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:32:06.901516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.901493 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-xks-ca" pod "c03a00fe-cc7e-4d1b-84d7-b45ed81901df" (UID: "c03a00fe-cc7e-4d1b-84d7-b45ed81901df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:06.901919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:06.901616 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-kube-api-access-sw27b" pod "c03a00fe-cc7e-4d1b-84d7-b45ed81901df" (UID: "c03a00fe-cc7e-4d1b-84d7-b45ed81901df"). InnerVolumeSpecName "kube-api-access-sw27b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:07.002186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.001410 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw27b\" (UniqueName: \"kubernetes.io/projected/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-kube-api-access-sw27b\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:32:07.002186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.001451 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03a00fe-cc7e-4d1b-84d7-b45ed81901df-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:32:07.570772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.570741 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d"] Nov 03 22:32:07.571560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.571544 4903 cpu_manager.go:414] "containerMap: removing container" podUID="cdd935e3-a795-44e5-a570-39a00ef9bbaa" containerName="revoke-keys" Nov 03 22:32:07.571613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.571562 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd935e3-a795-44e5-a570-39a00ef9bbaa" containerName="revoke-keys" Nov 03 22:32:07.571613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.571589 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c03a00fe-cc7e-4d1b-84d7-b45ed81901df" containerName="revoke-keys" Nov 03 22:32:07.571613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.571594 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03a00fe-cc7e-4d1b-84d7-b45ed81901df" containerName="revoke-keys" Nov 03 22:32:07.571712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.571700 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03a00fe-cc7e-4d1b-84d7-b45ed81901df" containerName="revoke-keys" Nov 03 22:32:07.571743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.571716 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd935e3-a795-44e5-a570-39a00ef9bbaa" containerName="revoke-keys" Nov 03 22:32:07.575574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.575555 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.593778 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.593751 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d"] Nov 03 22:32:07.612791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.612760 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff769914-svw6d\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.612957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.612857 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff769914-svw6d\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.613011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.612992 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j82sj\" (UniqueName: \"kubernetes.io/projected/53abd825-5969-4f59-9119-9b0abdc7d74b-kube-api-access-j82sj\") pod \"maas-api-revoke-sub-ff769914-svw6d\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.665573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.665545 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" Nov 03 22:32:07.665961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.665584 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f" event={"ID":"c03a00fe-cc7e-4d1b-84d7-b45ed81901df","Type":"ContainerDied","Data":"a316e69fb042ce81c8f84ae23c53a944fa08b7a22ee83d5e53f2913439d86a22"} Nov 03 22:32:07.665961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.665638 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a316e69fb042ce81c8f84ae23c53a944fa08b7a22ee83d5e53f2913439d86a22" Nov 03 22:32:07.714205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.714175 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff769914-svw6d\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.714391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.714256 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff769914-svw6d\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.714391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.714383 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j82sj\" (UniqueName: \"kubernetes.io/projected/53abd825-5969-4f59-9119-9b0abdc7d74b-kube-api-access-j82sj\") pod \"maas-api-revoke-sub-ff769914-svw6d\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.715005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.714979 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff769914-svw6d\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.716630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.716609 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff769914-svw6d\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.736673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.736642 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82sj\" (UniqueName: \"kubernetes.io/projected/53abd825-5969-4f59-9119-9b0abdc7d74b-kube-api-access-j82sj\") pod \"maas-api-revoke-sub-ff769914-svw6d\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:07.893620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:07.893551 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:08.240619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:08.238478 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d"] Nov 03 22:32:08.675098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:08.675068 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" event={"ID":"53abd825-5969-4f59-9119-9b0abdc7d74b","Type":"ContainerStarted","Data":"cbf88e6852ffda45bfe34fb22483fa56c9df353ddd4dbbe91432815ebc958aef"} Nov 03 22:32:08.675479 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:08.675123 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" event={"ID":"53abd825-5969-4f59-9119-9b0abdc7d74b","Type":"ContainerStarted","Data":"f39e0c329e62a4dbcdf8e937e3109762ffacdce4a809e0a05d73349328eb0864"} Nov 03 22:32:08.715818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:08.715768 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" podStartSLOduration=1.715754495 podStartE2EDuration="1.715754495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:08.711825967 +0000 UTC m=+1155.115766943" watchObservedRunningTime="2026-11-03 22:32:08.715754495 +0000 UTC m=+1155.119695469" Nov 03 22:32:09.248890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.248839 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6"] Nov 03 22:32:09.254079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.254057 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.278239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.278216 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-0847aa-openshift-default-dockercfg-dkbjf\"" Nov 03 22:32:09.287352 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.287321 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6"] Nov 03 22:32:09.350286 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.350248 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b3f78-d76d-4677-8d92-8fc048cb9185-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.350729 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.350697 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.355400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.350771 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.355400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.350827 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skcqt\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-kube-api-access-skcqt\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.355400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.350894 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.355400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.350961 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.355400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.351013 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.355400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.351056 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.355400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.351093 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.453511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.453472 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b3f78-d76d-4677-8d92-8fc048cb9185-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.453687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.453606 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.453687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.453672 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.453853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.453726 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skcqt\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-kube-api-access-skcqt\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.453853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.453788 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.453853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.453845 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.454022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.453872 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.454022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.453902 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.454022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.453940 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.456774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.454493 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.456774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.454498 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b3f78-d76d-4677-8d92-8fc048cb9185-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.456774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.455041 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.456774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.455323 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.456774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.455488 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.458778 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.458758 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.459150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.459129 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.475810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.475780 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.476042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.476016 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skcqt\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-kube-api-access-skcqt\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.614028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.613947 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:09.689830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.689799 4903 generic.go:356] "Generic (PLEG): container finished" podID="53abd825-5969-4f59-9119-9b0abdc7d74b" containerID="cbf88e6852ffda45bfe34fb22483fa56c9df353ddd4dbbe91432815ebc958aef" exitCode=0 Nov 03 22:32:09.690251 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.690022 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" event={"ID":"53abd825-5969-4f59-9119-9b0abdc7d74b","Type":"ContainerDied","Data":"cbf88e6852ffda45bfe34fb22483fa56c9df353ddd4dbbe91432815ebc958aef"} Nov 03 22:32:09.938803 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:32:09.938764 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480b3f78_d76d_4677_8d92_8fc048cb9185.slice/crio-c342ac4035c2c040d9e40cf84650bfea94bde896d37b2f768c37bcf10da0847b WatchSource:0}: Error finding container c342ac4035c2c040d9e40cf84650bfea94bde896d37b2f768c37bcf10da0847b: Status 404 returned error can't find the container with id c342ac4035c2c040d9e40cf84650bfea94bde896d37b2f768c37bcf10da0847b Nov 03 22:32:09.942317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.942258 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:09.942437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.942369 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:09.942437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.942413 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:09.943132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:09.943107 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6"] Nov 03 22:32:10.700992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:10.700958 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" event={"ID":"480b3f78-d76d-4677-8d92-8fc048cb9185","Type":"ContainerStarted","Data":"bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d"} Nov 03 22:32:10.701413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:10.701016 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" event={"ID":"480b3f78-d76d-4677-8d92-8fc048cb9185","Type":"ContainerStarted","Data":"c342ac4035c2c040d9e40cf84650bfea94bde896d37b2f768c37bcf10da0847b"} Nov 03 22:32:10.775316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:10.774132 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" podStartSLOduration=1.774111538 podStartE2EDuration="1.774111538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:10.745934918 +0000 UTC m=+1157.149875897" watchObservedRunningTime="2026-11-03 22:32:10.774111538 +0000 UTC m=+1157.178052515" Nov 03 22:32:11.088471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.088401 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:11.118994 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.118954 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-xks-ca\") pod \"53abd825-5969-4f59-9119-9b0abdc7d74b\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " Nov 03 22:32:11.119154 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.119141 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53abd825-5969-4f59-9119-9b0abdc7d74b-kube-api-access-j82sj\" (UniqueName: \"kubernetes.io/projected/53abd825-5969-4f59-9119-9b0abdc7d74b-kube-api-access-j82sj\") pod \"53abd825-5969-4f59-9119-9b0abdc7d74b\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " Nov 03 22:32:11.119261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.119245 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-openshift-ca\") pod \"53abd825-5969-4f59-9119-9b0abdc7d74b\" (UID: \"53abd825-5969-4f59-9119-9b0abdc7d74b\") " Nov 03 22:32:11.120550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.120521 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-openshift-ca" pod "53abd825-5969-4f59-9119-9b0abdc7d74b" (UID: "53abd825-5969-4f59-9119-9b0abdc7d74b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:32:11.146355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.134838 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53abd825-5969-4f59-9119-9b0abdc7d74b-kube-api-access-j82sj" pod "53abd825-5969-4f59-9119-9b0abdc7d74b" (UID: "53abd825-5969-4f59-9119-9b0abdc7d74b"). InnerVolumeSpecName "kube-api-access-j82sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:32:11.146355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.136568 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-xks-ca" pod "53abd825-5969-4f59-9119-9b0abdc7d74b" (UID: "53abd825-5969-4f59-9119-9b0abdc7d74b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:32:11.222611 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.222524 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:32:11.222611 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.222550 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j82sj\" (UniqueName: \"kubernetes.io/projected/53abd825-5969-4f59-9119-9b0abdc7d74b-kube-api-access-j82sj\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:32:11.222611 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.222562 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53abd825-5969-4f59-9119-9b0abdc7d74b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:32:11.614963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.614929 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:11.617176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.617148 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 22:32:11.617338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.617197 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 22:32:11.711426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.711389 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" Nov 03 22:32:11.711927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.711382 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d" event={"ID":"53abd825-5969-4f59-9119-9b0abdc7d74b","Type":"ContainerDied","Data":"f39e0c329e62a4dbcdf8e937e3109762ffacdce4a809e0a05d73349328eb0864"} Nov 03 22:32:11.712049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:11.711960 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f39e0c329e62a4dbcdf8e937e3109762ffacdce4a809e0a05d73349328eb0864" Nov 03 22:32:12.614820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:12.614789 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 22:32:12.614982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:12.614842 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 22:32:13.614446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:13.614408 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 22:32:13.614830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:13.614483 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 22:32:13.617023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:13.617002 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 22:32:14.614881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:14.614845 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 22:32:14.615267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:14.614913 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 22:32:14.629268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:14.629239 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 22:32:15.038206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:15.038169 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp"] Nov 03 22:32:15.067715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:15.060205 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57273d5a-jdkqp"] Nov 03 22:32:15.619327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:15.617402 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 22:32:15.619327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:15.617464 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 22:32:15.737799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:15.737764 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2234ec0-095f-41f3-87ea-2040f0fac56c" path="/var/lib/kubelet/pods/c2234ec0-095f-41f3-87ea-2040f0fac56c/volumes" Nov 03 22:32:16.621081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:16.621052 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:16.621623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:16.621542 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:16.622553 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:16.622532 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:32:18.394611 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:32:18.394571 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:32:20.031712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:20.031678 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2"] Nov 03 22:32:20.041709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:20.041675 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ba2b51d-q2lf2"] Nov 03 22:32:21.738533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:21.738492 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae97f1e-444f-45a7-aef6-652078cdb910" path="/var/lib/kubelet/pods/dae97f1e-444f-45a7-aef6-652078cdb910/volumes" Nov 03 22:32:28.043375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:28.043335 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4"] Nov 03 22:32:28.051138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:28.051098 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7718741e-6s6m4"] Nov 03 22:32:29.737541 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:29.737503 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0a4ac4-1a34-441c-a4d5-cd5c42b64758" path="/var/lib/kubelet/pods/0c0a4ac4-1a34-441c-a4d5-cd5c42b64758/volumes" Nov 03 22:32:35.841795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:35.841762 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:32:36.049244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:36.049193 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2"] Nov 03 22:32:36.057105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:36.057072 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b24c65aa-p86c2"] Nov 03 22:32:36.660327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:36.649839 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f"] Nov 03 22:32:36.694842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:36.694801 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-53879c92-5vd6f"] Nov 03 22:32:37.785659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:37.785610 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15627fc7-38ee-4869-b2d5-6bcd7f9300a0" path="/var/lib/kubelet/pods/15627fc7-38ee-4869-b2d5-6bcd7f9300a0/volumes" Nov 03 22:32:37.786983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:37.786930 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03a00fe-cc7e-4d1b-84d7-b45ed81901df" path="/var/lib/kubelet/pods/c03a00fe-cc7e-4d1b-84d7-b45ed81901df/volumes" Nov 03 22:32:39.035076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:39.035034 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj"] Nov 03 22:32:39.068665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:39.068625 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2017624a-fjjvj"] Nov 03 22:32:39.741537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:39.741504 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873" path="/var/lib/kubelet/pods/4e8d38b9-0d9c-4a78-bab7-18ad9c4bb873/volumes" Nov 03 22:32:46.060955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.060921 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk"] Nov 03 22:32:46.063525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.061809 4903 cpu_manager.go:414] "containerMap: removing container" podUID="53abd825-5969-4f59-9119-9b0abdc7d74b" containerName="revoke-keys" Nov 03 22:32:46.063525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.061824 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53abd825-5969-4f59-9119-9b0abdc7d74b" containerName="revoke-keys" Nov 03 22:32:46.063525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.061944 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="53abd825-5969-4f59-9119-9b0abdc7d74b" containerName="revoke-keys" Nov 03 22:32:46.065694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.065675 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.092324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.092278 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk"] Nov 03 22:32:46.126740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.126704 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-token\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.126894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.126747 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-data\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.126894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.126825 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.126970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.126951 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmzkl\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-kube-api-access-pmzkl\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.127005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.126972 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-podinfo\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.127005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.127000 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-credential-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.127174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.127154 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9847228-3a90-4625-a6b0-e17b44bed10e-istiod-ca-cert\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.127256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.127243 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-envoy\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.127311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.127278 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-certs\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.149242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.149212 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-23d281-openshift-default-dockercfg-88spn\"" Nov 03 22:32:46.228706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.228662 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-credential-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.228896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.228766 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9847228-3a90-4625-a6b0-e17b44bed10e-istiod-ca-cert\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.228896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.228835 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-envoy\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.228896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.228866 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-certs\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.228914 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-token\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.228946 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-data\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.228994 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.229188 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmzkl\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-kube-api-access-pmzkl\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.229222 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-podinfo\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.229252 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-certs\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.229551 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-credential-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.229585 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-data\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.229636 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-socket\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.229848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.229768 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9847228-3a90-4625-a6b0-e17b44bed10e-istiod-ca-cert\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.231656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.231632 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-envoy\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.231815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.231799 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-podinfo\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.251074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.251042 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-token\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.251231 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.251216 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmzkl\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-kube-api-access-pmzkl\") pod \"e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.384952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.384866 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:46.779121 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:32:46.779090 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9847228_3a90_4625_a6b0_e17b44bed10e.slice/crio-7eba6bddb98b0daf40ae0980c15c69a4d6e8923453dc4d44959d7d8859346150 WatchSource:0}: Error finding container 7eba6bddb98b0daf40ae0980c15c69a4d6e8923453dc4d44959d7d8859346150: Status 404 returned error can't find the container with id 7eba6bddb98b0daf40ae0980c15c69a4d6e8923453dc4d44959d7d8859346150 Nov 03 22:32:46.784440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.784409 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk"] Nov 03 22:32:46.788941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.788904 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:46.789068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.788990 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:46.789068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:46.789033 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:47.212925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.212881 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" event={"ID":"f9847228-3a90-4625-a6b0-e17b44bed10e","Type":"ContainerStarted","Data":"997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d"} Nov 03 22:32:47.213477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.212953 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" event={"ID":"f9847228-3a90-4625-a6b0-e17b44bed10e","Type":"ContainerStarted","Data":"7eba6bddb98b0daf40ae0980c15c69a4d6e8923453dc4d44959d7d8859346150"} Nov 03 22:32:47.288416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.288341 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" podStartSLOduration=1.288318763 podStartE2EDuration="1.288318763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:47.268929093 +0000 UTC m=+1193.672870082" watchObservedRunningTime="2026-11-03 22:32:47.288318763 +0000 UTC m=+1193.692259739" Nov 03 22:32:47.385950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.385913 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:47.393648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.393618 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:47.394878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.394848 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7"] Nov 03 22:32:47.401155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.401132 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.417582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.417546 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8b1efeb6-openshift-default-dockercfg-c2bhg\"" Nov 03 22:32:47.422174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.422144 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7"] Nov 03 22:32:47.450103 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.450069 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-certs\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.450340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.450109 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgwdg\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-kube-api-access-jgwdg\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.450340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.450206 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f335df24-eab6-49f0-a307-86fea13f9402-istio-podinfo\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.450500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.450378 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-data\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.450500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.450479 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.450596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.450507 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f335df24-eab6-49f0-a307-86fea13f9402-istiod-ca-cert\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.450596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.450553 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-istio-token\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.450681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.450645 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-envoy\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.450741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.450725 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-credential-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.554989 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-data\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.555084 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.555111 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f335df24-eab6-49f0-a307-86fea13f9402-istiod-ca-cert\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.555155 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-istio-token\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.555271 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-envoy\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.555372 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-credential-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.555530 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-certs\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.555562 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgwdg\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-kube-api-access-jgwdg\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.555629 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f335df24-eab6-49f0-a307-86fea13f9402-istio-podinfo\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.556555 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-data\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.557366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.556809 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.558379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.557534 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f335df24-eab6-49f0-a307-86fea13f9402-istiod-ca-cert\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.558379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.557792 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-credential-socket\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.558498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.558411 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-certs\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.571058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.571030 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f335df24-eab6-49f0-a307-86fea13f9402-istio-podinfo\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.571694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.571674 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-envoy\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.577647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.577618 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-istio-token\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.579650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.579625 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgwdg\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-kube-api-access-jgwdg\") pod \"e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:47.729515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:47.729481 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:48.068670 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:32:48.068634 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf335df24_eab6_49f0_a307_86fea13f9402.slice/crio-a59e0f7f21a0da5622538cf74d5923bb1f3cb495d315bfa1b40d6228c4b10caf WatchSource:0}: Error finding container a59e0f7f21a0da5622538cf74d5923bb1f3cb495d315bfa1b40d6228c4b10caf: Status 404 returned error can't find the container with id a59e0f7f21a0da5622538cf74d5923bb1f3cb495d315bfa1b40d6228c4b10caf Nov 03 22:32:48.070529 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.070452 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7"] Nov 03 22:32:48.071519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.071486 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:48.071610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.071553 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:48.071610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.071582 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:48.223956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.223917 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" event={"ID":"f335df24-eab6-49f0-a307-86fea13f9402","Type":"ContainerStarted","Data":"58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a"} Nov 03 22:32:48.224499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.223974 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" event={"ID":"f335df24-eab6-49f0-a307-86fea13f9402","Type":"ContainerStarted","Data":"a59e0f7f21a0da5622538cf74d5923bb1f3cb495d315bfa1b40d6228c4b10caf"} Nov 03 22:32:48.224499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.224235 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:48.225410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.225387 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:32:48.273811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.273761 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" podStartSLOduration=1.273747264 podStartE2EDuration="1.273747264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:48.26536565 +0000 UTC m=+1194.669306625" watchObservedRunningTime="2026-11-03 22:32:48.273747264 +0000 UTC m=+1194.677688238" Nov 03 22:32:48.389959 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:32:48.389876 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:32:48.729793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:48.729760 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:49.737210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:49.737182 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:50.035632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:50.035537 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw"] Nov 03 22:32:50.041017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:50.040987 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29584f7e-nzxdw"] Nov 03 22:32:50.254738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:50.254709 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:50.255753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:50.255732 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:32:51.745279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:51.745242 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a0857d-fdfa-4ac7-a969-8d08d87aeac0" path="/var/lib/kubelet/pods/d5a0857d-fdfa-4ac7-a969-8d08d87aeac0/volumes" Nov 03 22:32:53.962492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:53.962460 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:32:53.962925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:53.962460 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:32:53.973549 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:53.973521 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:32:53.973747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:53.973523 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:32:54.480787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.480753 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl"] Nov 03 22:32:54.490725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.490652 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.520043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.520006 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.520358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.520302 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.520459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.520360 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.520459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.520392 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tql\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-kube-api-access-r9tql\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.520459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.520436 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.520622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.520607 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.520680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.520654 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.520738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.520684 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.520738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.520722 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.659397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.633369 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.659795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.659759 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.660007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.659989 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.660127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.660111 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tql\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-kube-api-access-r9tql\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.660256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.660243 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.660536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.660521 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.660673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.660658 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.660774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.660749 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.661015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.660997 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.661931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.661860 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.662502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.662478 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.663265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.663243 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.670201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.665803 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.670201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.666047 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.712477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.691599 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tql\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-kube-api-access-r9tql\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.729987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.729947 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8"] Nov 03 22:32:54.747471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.734172 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.747471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.741974 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.747471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.745359 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.799340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.792362 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl"] Nov 03 22:32:54.799340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.792523 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.814372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.811448 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8"] Nov 03 22:32:54.867083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.867043 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.867248 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.867110 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.867248 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.867178 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.867368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.867256 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.867419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.867392 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.867456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.867439 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.867551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.867532 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjsmg\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-kube-api-access-xjsmg\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.867596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.867582 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.867643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.867628 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.910271 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.910235 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:54.969288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969254 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.969763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969327 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.969763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969384 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjsmg\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-kube-api-access-xjsmg\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.969763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969425 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.969763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969446 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.969763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969568 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.969763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969604 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.969763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969653 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.969763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969682 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.969763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.969719 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-data\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.970237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.970118 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-credential-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.970237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.970134 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-socket\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.970237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.970198 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-certs\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.970495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.970470 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istiod-ca-cert\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.972804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.972579 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-envoy\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.973092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.973071 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-podinfo\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.990857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.990830 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-token\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:54.991328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:54.991264 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjsmg\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-kube-api-access-xjsmg\") pod \"e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:55.048574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.048495 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd"] Nov 03 22:32:55.059915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.059871 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19feee9a-d5cqd"] Nov 03 22:32:55.118107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.118079 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:55.260032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.260000 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl"] Nov 03 22:32:55.265564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.265533 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:55.265647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.265600 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:55.265647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.265631 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:55.316189 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.316156 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" event={"ID":"d560378b-a5b9-4a00-8f7c-7319c1d1bb87","Type":"ContainerStarted","Data":"6995fceaf6b63bf645ebabda487a4c848e088b4e390c6bf1b2b905353ea0ee28"} Nov 03 22:32:55.438258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.438226 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff"] Nov 03 22:32:55.443382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.443355 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.452709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.452677 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8"] Nov 03 22:32:55.458664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.458628 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:55.459452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.459428 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:55.459618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.459602 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:55.478120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.478087 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.478256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.478147 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.478256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.478239 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.478407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.478287 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.478464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.478420 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1395a56f-875d-4faa-a8a6-c1caf5438efe-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.478520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.478459 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.478948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.478572 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjj7\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-kube-api-access-qxjj7\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.478948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.478687 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.478948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.478793 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.527309 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.527269 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff"] Nov 03 22:32:55.536516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.536485 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps"] Nov 03 22:32:55.539094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.539044 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a940d5-openshift-default-dockercfg-9b7lq\"" Nov 03 22:32:55.544444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.544420 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps"] Nov 03 22:32:55.544556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.544535 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.582582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.582546 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.582745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.582614 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.582745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.582674 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9865da23-stbps\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.582877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.582752 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1395a56f-875d-4faa-a8a6-c1caf5438efe-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.582877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.582784 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.582991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.582936 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjj7\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-kube-api-access-qxjj7\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.582991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.582983 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4pf\" (UniqueName: \"kubernetes.io/projected/253fa807-c223-4e87-8275-dc76cac0df01-kube-api-access-6x4pf\") pod \"maas-api-revoke-sub-9865da23-stbps\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.583106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.583070 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.583160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.583111 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.583160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.583116 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9865da23-stbps\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.583872 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.583845 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1395a56f-875d-4faa-a8a6-c1caf5438efe-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.584006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.583961 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.584070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.584042 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.584128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.584069 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.584128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.584082 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.584468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.584446 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.584586 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.584468 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.587137 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.587114 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.587309 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.587184 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.608511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.608484 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:32:55.609543 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.609517 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:32:55.652152 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.652090 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:32:55.652389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.652356 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:32:55.672059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.672024 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjj7\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-kube-api-access-qxjj7\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.673035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.672987 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.685755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.685723 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9865da23-stbps\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.685936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.685919 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9865da23-stbps\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.686106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.686076 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4pf\" (UniqueName: \"kubernetes.io/projected/253fa807-c223-4e87-8275-dc76cac0df01-kube-api-access-6x4pf\") pod \"maas-api-revoke-sub-9865da23-stbps\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.686549 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.686527 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9865da23-stbps\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.688843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.688571 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9865da23-stbps\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.724053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.724018 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4pf\" (UniqueName: \"kubernetes.io/projected/253fa807-c223-4e87-8275-dc76cac0df01-kube-api-access-6x4pf\") pod \"maas-api-revoke-sub-9865da23-stbps\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.737564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.737526 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e54eaf8-3ac0-44c2-8a4c-41c26432067f" path="/var/lib/kubelet/pods/8e54eaf8-3ac0-44c2-8a4c-41c26432067f/volumes" Nov 03 22:32:55.798110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.798076 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:55.886552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.886523 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:32:55.936510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:55.936382 4903 scope.go:122] "RemoveContainer" containerID="120db8bf582bc3e9fb95ae1050b84f285025e446181d879a8fcc936f627d6e59" Nov 03 22:32:56.103386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.103338 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf"] Nov 03 22:32:56.111341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.111303 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a43e5677-fjmvf"] Nov 03 22:32:56.149317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.147057 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff"] Nov 03 22:32:56.174912 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:32:56.159006 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1395a56f_875d_4faa_a8a6_c1caf5438efe.slice/crio-8f02afd199c6e329b238d89866454c6ca78e49dda9d527034f1e9022487f4a39 WatchSource:0}: Error finding container 8f02afd199c6e329b238d89866454c6ca78e49dda9d527034f1e9022487f4a39: Status 404 returned error can't find the container with id 8f02afd199c6e329b238d89866454c6ca78e49dda9d527034f1e9022487f4a39 Nov 03 22:32:56.174912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.163890 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:56.174912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.163980 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:56.174912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.164025 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:32:56.536343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.536306 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps"] Nov 03 22:32:56.539456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.539179 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" event={"ID":"1395a56f-875d-4faa-a8a6-c1caf5438efe","Type":"ContainerStarted","Data":"8f02afd199c6e329b238d89866454c6ca78e49dda9d527034f1e9022487f4a39"} Nov 03 22:32:56.605710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.605661 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" event={"ID":"98d3f93d-025c-4aa4-88f1-af33c03fbb13","Type":"ContainerStarted","Data":"55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e"} Nov 03 22:32:56.605933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.605916 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" event={"ID":"98d3f93d-025c-4aa4-88f1-af33c03fbb13","Type":"ContainerStarted","Data":"54fc8f586281e1bd25859ec8a58baab2fa706de5c660a699fc227b4043d0c259"} Nov 03 22:32:56.614162 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.614102 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" event={"ID":"d560378b-a5b9-4a00-8f7c-7319c1d1bb87","Type":"ContainerStarted","Data":"c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874"} Nov 03 22:32:56.702948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.687510 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" podStartSLOduration=2.687487979 podStartE2EDuration="2.687487979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:56.686840409 +0000 UTC m=+1203.090781397" watchObservedRunningTime="2026-11-03 22:32:56.687487979 +0000 UTC m=+1203.091428957" Nov 03 22:32:56.732093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.732034 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" podStartSLOduration=2.732015659 podStartE2EDuration="2.732015659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:56.729145342 +0000 UTC m=+1203.133086322" watchObservedRunningTime="2026-11-03 22:32:56.732015659 +0000 UTC m=+1203.135956634" Nov 03 22:32:56.910666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.910629 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:32:56.911620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.911593 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:32:56.911759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:56.911638 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:32:57.119756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:57.119515 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:32:57.120125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:57.120069 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:32:57.120125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:57.120113 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:32:57.939681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:57.939014 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:32:57.939971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:57.939939 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:32:58.125883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.121941 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:32:58.125883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.122017 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:32:58.230347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.200500 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20505c61-dd5a-4fb9-81f2-25450e159897" path="/var/lib/kubelet/pods/20505c61-dd5a-4fb9-81f2-25450e159897/volumes" Nov 03 22:32:58.230347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.201080 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" event={"ID":"1395a56f-875d-4faa-a8a6-c1caf5438efe","Type":"ContainerStarted","Data":"32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561"} Nov 03 22:32:58.230347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.201139 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" event={"ID":"253fa807-c223-4e87-8275-dc76cac0df01","Type":"ContainerStarted","Data":"12b38477bab24acec66949a4b5ef37600a8c5b1c1caa99945015e86f1cb9d21d"} Nov 03 22:32:58.230347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.201165 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" event={"ID":"253fa807-c223-4e87-8275-dc76cac0df01","Type":"ContainerStarted","Data":"8263c10a2de12db679ffca6b2ca76ad5d73d38beed1dd311f99dbca321579bbf"} Nov 03 22:32:58.329281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.241891 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" podStartSLOduration=3.241870314 podStartE2EDuration="3.241870314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:58.187411885 +0000 UTC m=+1204.591352857" watchObservedRunningTime="2026-11-03 22:32:58.241870314 +0000 UTC m=+1204.645811294" Nov 03 22:32:58.812321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.806479 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:32:58.824318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.823460 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:32:58.824318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.823525 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 22:32:58.895424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.885176 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" podStartSLOduration=3.885152373 podStartE2EDuration="3.885152373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:32:58.542493167 +0000 UTC m=+1204.946434133" watchObservedRunningTime="2026-11-03 22:32:58.885152373 +0000 UTC m=+1205.289093350" Nov 03 22:32:58.945321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.919966 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:32:58.945650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:58.945608 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:32:59.119369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:59.118609 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:32:59.119369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:59.118693 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:32:59.265795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:59.244774 4903 generic.go:356] "Generic (PLEG): container finished" podID="253fa807-c223-4e87-8275-dc76cac0df01" containerID="12b38477bab24acec66949a4b5ef37600a8c5b1c1caa99945015e86f1cb9d21d" exitCode=0 Nov 03 22:32:59.268412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:59.268370 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" event={"ID":"253fa807-c223-4e87-8275-dc76cac0df01","Type":"ContainerDied","Data":"12b38477bab24acec66949a4b5ef37600a8c5b1c1caa99945015e86f1cb9d21d"} Nov 03 22:32:59.807782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:59.807739 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:32:59.808090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:59.808055 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 22:32:59.917835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:59.917785 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:32:59.918131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:32:59.918095 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:33:00.127910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:00.126469 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:33:00.127910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:00.126541 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:33:00.861697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:00.856814 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:33:00.861697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:00.856869 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 22:33:01.129616 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.126816 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:33:01.129616 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.126896 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:33:01.129616 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.128045 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:33:01.129616 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.128090 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:33:01.254380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.253563 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:33:01.530402 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.502012 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-openshift-ca\") pod \"253fa807-c223-4e87-8275-dc76cac0df01\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " Nov 03 22:33:01.530402 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.502077 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-xks-ca\") pod \"253fa807-c223-4e87-8275-dc76cac0df01\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " Nov 03 22:33:01.530402 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.502283 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253fa807-c223-4e87-8275-dc76cac0df01-kube-api-access-6x4pf\" (UniqueName: \"kubernetes.io/projected/253fa807-c223-4e87-8275-dc76cac0df01-kube-api-access-6x4pf\") pod \"253fa807-c223-4e87-8275-dc76cac0df01\" (UID: \"253fa807-c223-4e87-8275-dc76cac0df01\") " Nov 03 22:33:01.530402 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.526775 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-openshift-ca" pod "253fa807-c223-4e87-8275-dc76cac0df01" (UID: "253fa807-c223-4e87-8275-dc76cac0df01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:33:01.769469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.744830 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r"] Nov 03 22:33:01.769469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.746369 4903 cpu_manager.go:414] "containerMap: removing container" podUID="253fa807-c223-4e87-8275-dc76cac0df01" containerName="revoke-keys" Nov 03 22:33:01.769469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.746404 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253fa807-c223-4e87-8275-dc76cac0df01" containerName="revoke-keys" Nov 03 22:33:01.769469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.746654 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="253fa807-c223-4e87-8275-dc76cac0df01" containerName="revoke-keys" Nov 03 22:33:01.832721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.831460 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:33:01.901318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.900408 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:33:01.901318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.900473 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 22:33:01.941532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:01.941494 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:02.030323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.027524 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-2a2eca-dockercfg-9lnlz\"" Nov 03 22:33:02.038318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.030576 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:33:02.038318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.030633 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:33:02.038318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.037921 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-xks-ca" pod "253fa807-c223-4e87-8275-dc76cac0df01" (UID: "253fa807-c223-4e87-8275-dc76cac0df01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:33:02.038318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.038046 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-2a2eca\"" Nov 03 22:33:02.046061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.046024 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253fa807-c223-4e87-8275-dc76cac0df01-kube-api-access-6x4pf" pod "253fa807-c223-4e87-8275-dc76cac0df01" (UID: "253fa807-c223-4e87-8275-dc76cac0df01"). InnerVolumeSpecName "kube-api-access-6x4pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:33:02.052014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.048815 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:02.054594 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.054566 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" Nov 03 22:33:02.065041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.065002 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xsqz\" (UniqueName: \"kubernetes.io/projected/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-kube-api-access-7xsqz\") pod \"payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r\" (UID: \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:02.090375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.085390 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:33:02.091059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.091019 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r\" (UID: \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:02.091539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.091515 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/253fa807-c223-4e87-8275-dc76cac0df01-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:33:02.091886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.091866 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x4pf\" (UniqueName: \"kubernetes.io/projected/253fa807-c223-4e87-8275-dc76cac0df01-kube-api-access-6x4pf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:33:02.120061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.120018 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 22:33:02.120429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.120397 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 22:33:02.137312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.135798 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps" event={"ID":"253fa807-c223-4e87-8275-dc76cac0df01","Type":"ContainerDied","Data":"8263c10a2de12db679ffca6b2ca76ad5d73d38beed1dd311f99dbca321579bbf"} Nov 03 22:33:02.137312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.135855 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8263c10a2de12db679ffca6b2ca76ad5d73d38beed1dd311f99dbca321579bbf" Nov 03 22:33:02.137312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.135875 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r"] Nov 03 22:33:02.195691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.194635 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r\" (UID: \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:02.211797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.210323 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r\" (UID: \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:02.211797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.210480 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xsqz\" (UniqueName: \"kubernetes.io/projected/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-kube-api-access-7xsqz\") pod \"payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r\" (UID: \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:02.236424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.236387 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xsqz\" (UniqueName: \"kubernetes.io/projected/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-kube-api-access-7xsqz\") pod \"payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r\" (UID: \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:02.367462 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.367079 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:02.780922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.780882 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r"] Nov 03 22:33:02.799041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.799003 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 22:33:02.799200 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.799069 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 22:33:02.917690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.917649 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:33:02.918439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.918415 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:33:02.919491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:02.919472 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:33:03.090622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.090533 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" event={"ID":"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d","Type":"ContainerStarted","Data":"d597d25a664cf0d3f1ad9bfe1d49f12080eae47ccf1f2e7dcdcc0ee9aa2b37ca"} Nov 03 22:33:03.090849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.090828 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" event={"ID":"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d","Type":"ContainerStarted","Data":"f2b6f9b18a22a567094cafb75dddec35f4539434cd8f7dfb4a251ec20146b08a"} Nov 03 22:33:03.090941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.090932 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:03.127613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.127577 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:33:03.128561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.128536 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:33:03.129006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.128982 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:33:03.130399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.130352 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" podStartSLOduration=2.130210858 podStartE2EDuration="2.130210858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:03.127382084 +0000 UTC m=+1209.531323059" watchObservedRunningTime="2026-11-03 22:33:03.130210858 +0000 UTC m=+1209.534151832" Nov 03 22:33:03.804171 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.804140 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:33:03.804562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.804535 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:33:03.805624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:03.805594 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:33:05.037990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:05.037958 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89"] Nov 03 22:33:05.047983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:05.047950 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133efb01-pjm89"] Nov 03 22:33:05.737906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:05.737870 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465" path="/var/lib/kubelet/pods/5e0ab02e-7a6c-4fe3-9c3f-fb2160e25465/volumes" Nov 03 22:33:06.036039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:06.035879 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl"] Nov 03 22:33:06.048927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:06.048886 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4360e7-mnhwl"] Nov 03 22:33:07.737854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:07.737818 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf0abb6-faaa-4246-8586-ba38c593260c" path="/var/lib/kubelet/pods/8cf0abb6-faaa-4246-8586-ba38c593260c/volumes" Nov 03 22:33:10.032492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:10.032457 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj"] Nov 03 22:33:10.043242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:10.043207 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx"] Nov 03 22:33:10.051972 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:10.051935 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f6e76e4-d2jwj"] Nov 03 22:33:10.057326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:10.057286 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88290bcd-vgxhx"] Nov 03 22:33:11.739476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:11.739429 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812" path="/var/lib/kubelet/pods/1b0e9f0f-37bf-4d76-ab40-8e1b1d1cf812/volumes" Nov 03 22:33:11.740151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:11.740133 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d75dc8-035d-4aed-a647-b878a5f93536" path="/var/lib/kubelet/pods/74d75dc8-035d-4aed-a647-b878a5f93536/volumes" Nov 03 22:33:14.031025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:14.030991 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g"] Nov 03 22:33:14.040786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:14.040759 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e772e-r4g7g"] Nov 03 22:33:14.104219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:14.104193 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 22:33:15.742193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:15.742149 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f83a94-43bc-4b3d-9251-2b5e53545371" path="/var/lib/kubelet/pods/f7f83a94-43bc-4b3d-9251-2b5e53545371/volumes" Nov 03 22:33:16.036942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:16.036847 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t"] Nov 03 22:33:16.047454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:16.047374 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72c32694-ns97t"] Nov 03 22:33:17.738036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:17.737998 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0cbee8-b2b0-4fac-8752-77c9853c2015" path="/var/lib/kubelet/pods/3a0cbee8-b2b0-4fac-8752-77c9853c2015/volumes" Nov 03 22:33:18.385466 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:33:18.385420 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:33:20.067317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:20.067208 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92"] Nov 03 22:33:20.075233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:20.075199 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ae46c7e-vlh92"] Nov 03 22:33:21.798323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:21.798205 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b19000-c44c-4a8e-a84e-b36b9f826e2a" path="/var/lib/kubelet/pods/d3b19000-c44c-4a8e-a84e-b36b9f826e2a/volumes" Nov 03 22:33:25.036923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:25.036881 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw"] Nov 03 22:33:25.047469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:25.047426 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b92ca3d4-j7pzw"] Nov 03 22:33:25.740203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:25.740169 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a3ed9e-5d91-4625-8740-f98759711b30" path="/var/lib/kubelet/pods/f3a3ed9e-5d91-4625-8740-f98759711b30/volumes" Nov 03 22:33:28.030374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:28.030334 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647"] Nov 03 22:33:28.040683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:28.040653 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811e2270-sm647"] Nov 03 22:33:29.737209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:29.737175 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a44ef58-b8fe-4312-9e9b-04dd91e05b0d" path="/var/lib/kubelet/pods/6a44ef58-b8fe-4312-9e9b-04dd91e05b0d/volumes" Nov 03 22:33:30.031340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:30.031244 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc"] Nov 03 22:33:30.041059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:30.041027 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73db1173-cjdzc"] Nov 03 22:33:31.738532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:31.738502 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31b82496-2c18-435b-a7af-9aa1e8cdec24" path="/var/lib/kubelet/pods/31b82496-2c18-435b-a7af-9aa1e8cdec24/volumes" Nov 03 22:33:40.031966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.031927 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d"] Nov 03 22:33:40.041645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.041621 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed0a4a3-d7c2d"] Nov 03 22:33:40.617188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.617150 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k"] Nov 03 22:33:40.624216 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.624189 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.657314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.657263 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k"] Nov 03 22:33:40.706860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.706822 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.707071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.706878 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9993f4-b036-42b2-a58c-fe4264986bf9-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.707071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.707041 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.707202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.707176 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.707391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.707367 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.707473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.707441 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.707552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.707533 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcr4\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-kube-api-access-7gcr4\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.707615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.707573 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.707738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.707716 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.809519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.809480 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.809704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.809546 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.809704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.809619 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcr4\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-kube-api-access-7gcr4\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.809704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.809655 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.809874 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.809773 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.809940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.809919 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.810002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.809948 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9993f4-b036-42b2-a58c-fe4264986bf9-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.810104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.810082 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.810176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.810121 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-credential-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.810242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.810189 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.810339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.810273 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-socket\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.810560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.810535 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-certs\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.810654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.810561 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-data\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.810734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.810712 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9993f4-b036-42b2-a58c-fe4264986bf9-istiod-ca-cert\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.812531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.812511 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-envoy\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.812738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.812718 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-podinfo\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.831142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.831119 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-token\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.831368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.831345 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcr4\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-kube-api-access-7gcr4\") pod \"e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:40.945898 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:40.945866 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:41.037269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.035710 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d"] Nov 03 22:33:41.042914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.042886 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-510e733b-lph9d"] Nov 03 22:33:41.278342 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:33:41.278272 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9993f4_b036_42b2_a58c_fe4264986bf9.slice/crio-1425468c355a16966b46b1283efaa93a5053b8e3263f92ee18208afaf4f757b5 WatchSource:0}: Error finding container 1425468c355a16966b46b1283efaa93a5053b8e3263f92ee18208afaf4f757b5: Status 404 returned error can't find the container with id 1425468c355a16966b46b1283efaa93a5053b8e3263f92ee18208afaf4f757b5 Nov 03 22:33:41.278916 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.278893 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k"] Nov 03 22:33:41.281487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.281456 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:33:41.281577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.281520 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:33:41.281577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.281550 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:33:41.831313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.830804 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27a4e00-abaf-47bb-a3d3-6f2f59035d1c" path="/var/lib/kubelet/pods/f27a4e00-abaf-47bb-a3d3-6f2f59035d1c/volumes" Nov 03 22:33:41.842812 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.831639 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f65bce2e-b44d-46cc-8e78-b7de172f8611" path="/var/lib/kubelet/pods/f65bce2e-b44d-46cc-8e78-b7de172f8611/volumes" Nov 03 22:33:41.842812 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.832145 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" event={"ID":"ab9993f4-b036-42b2-a58c-fe4264986bf9","Type":"ContainerStarted","Data":"03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409"} Nov 03 22:33:41.842812 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.832219 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" event={"ID":"ab9993f4-b036-42b2-a58c-fe4264986bf9","Type":"ContainerStarted","Data":"1425468c355a16966b46b1283efaa93a5053b8e3263f92ee18208afaf4f757b5"} Nov 03 22:33:41.864773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.864723 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" podStartSLOduration=1.864708787 podStartE2EDuration="1.864708787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:41.860423778 +0000 UTC m=+1248.264364755" watchObservedRunningTime="2026-11-03 22:33:41.864708787 +0000 UTC m=+1248.268649761" Nov 03 22:33:41.954055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:41.947382 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:42.967013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:42.966429 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:33:42.967013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:42.966499 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 22:33:43.957349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:43.949711 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:33:43.957349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:43.949776 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 22:33:44.961222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:44.961186 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:33:44.961894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:44.961861 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 22:33:45.075965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:45.075927 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2"] Nov 03 22:33:45.083724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:45.083693 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076157af-jc5k2"] Nov 03 22:33:45.736225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:45.736192 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe4ad9f-4eb1-486d-8de3-9024890fde1a" path="/var/lib/kubelet/pods/bbe4ad9f-4eb1-486d-8de3-9024890fde1a/volumes" Nov 03 22:33:45.966317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:45.965440 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:33:45.966317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:45.965510 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 22:33:46.967110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:46.967064 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 22:33:46.967615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:46.967127 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 22:33:47.963220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:47.962485 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:47.963220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:47.963136 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:47.964181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:47.964147 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:33:48.262660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.262580 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t"] Nov 03 22:33:48.267527 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.267507 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:33:48.284121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.284095 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d8a3d1-dockercfg-s9kjb\"" Nov 03 22:33:48.287635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.287602 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d8a3d1\"" Nov 03 22:33:48.288933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.288910 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t"] Nov 03 22:33:48.407435 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.407390 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkhjk\" (UniqueName: \"kubernetes.io/projected/fddc0ee5-3581-450e-b82a-ca28c12cd15e-kube-api-access-pkhjk\") pod \"payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t\" (UID: \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:33:48.407779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.407749 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc0ee5-3581-450e-b82a-ca28c12cd15e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t\" (UID: \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:33:48.509652 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.509611 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkhjk\" (UniqueName: \"kubernetes.io/projected/fddc0ee5-3581-450e-b82a-ca28c12cd15e-kube-api-access-pkhjk\") pod \"payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t\" (UID: \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:33:48.510400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.510374 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc0ee5-3581-450e-b82a-ca28c12cd15e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t\" (UID: \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:33:48.525417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.521317 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc0ee5-3581-450e-b82a-ca28c12cd15e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t\" (UID: \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:33:48.539312 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:33:48.536685 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:33:48.543614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.543586 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkhjk\" (UniqueName: \"kubernetes.io/projected/fddc0ee5-3581-450e-b82a-ca28c12cd15e-kube-api-access-pkhjk\") pod \"payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t\" (UID: \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:33:48.588675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.588642 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:33:48.946947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:48.946916 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t"] Nov 03 22:33:48.947787 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:33:48.947763 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfddc0ee5_3581_450e_b82a_ca28c12cd15e.slice/crio-d399ce6d9f7b08cbf4feb8e96f98b26340faf2b35fa5e0c6e2f624f56d629348 WatchSource:0}: Error finding container d399ce6d9f7b08cbf4feb8e96f98b26340faf2b35fa5e0c6e2f624f56d629348: Status 404 returned error can't find the container with id d399ce6d9f7b08cbf4feb8e96f98b26340faf2b35fa5e0c6e2f624f56d629348 Nov 03 22:33:49.002735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.002698 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq"] Nov 03 22:33:49.010916 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.010888 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.027621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.024504 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq"] Nov 03 22:33:49.027621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.024762 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8b1efeb6\"/\"kube-root-ca.crt\"" Nov 03 22:33:49.093403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.093375 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8b1efeb6\"/\"default-dockercfg-cvm9h\"" Nov 03 22:33:49.093403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.093381 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8b1efeb6\"/\"e2e-lifecycle-model-8b1efeb6-kserve-self-signed-certs\"" Nov 03 22:33:49.103635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.103585 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8b1efeb6\"/\"openshift-service-ca.crt\"" Nov 03 22:33:49.138083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.138053 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-home\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.138261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.138090 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-tmp-dir\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.138350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.138282 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxrm\" (UniqueName: \"kubernetes.io/projected/e7602f85-77a4-4ed5-991f-08ad57b49be3-kube-api-access-nnxrm\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.138635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.138614 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-model-cache\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.138699 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.138653 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-dshm\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.138699 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.138681 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7602f85-77a4-4ed5-991f-08ad57b49be3-tls-certs\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.241245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.241151 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-model-cache\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.241245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.241209 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-dshm\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.241245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.241240 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7602f85-77a4-4ed5-991f-08ad57b49be3-tls-certs\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.241551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.241444 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-home\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.241551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.241475 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-tmp-dir\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.241658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.241572 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxrm\" (UniqueName: \"kubernetes.io/projected/e7602f85-77a4-4ed5-991f-08ad57b49be3-kube-api-access-nnxrm\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.241658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.241610 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-model-cache\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.242112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.242088 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-home\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.244679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.244657 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7602f85-77a4-4ed5-991f-08ad57b49be3-tls-certs\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.245038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.245021 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-dshm\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.246140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.246115 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-tmp-dir\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.289034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.289006 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxrm\" (UniqueName: \"kubernetes.io/projected/e7602f85-77a4-4ed5-991f-08ad57b49be3-kube-api-access-nnxrm\") pod \"e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.348430 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.341569 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:49.869688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.869658 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq"] Nov 03 22:33:49.936107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.936067 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" event={"ID":"e7602f85-77a4-4ed5-991f-08ad57b49be3","Type":"ContainerStarted","Data":"cea7c340bbf3792aedd3b9a69d7239611f9e46e118614525f4a901b37ea180e0"} Nov 03 22:33:49.949810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.947114 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" event={"ID":"fddc0ee5-3581-450e-b82a-ca28c12cd15e","Type":"ContainerStarted","Data":"5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30"} Nov 03 22:33:49.949810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.947184 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" event={"ID":"fddc0ee5-3581-450e-b82a-ca28c12cd15e","Type":"ContainerStarted","Data":"d399ce6d9f7b08cbf4feb8e96f98b26340faf2b35fa5e0c6e2f624f56d629348"} Nov 03 22:33:49.949810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.947714 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:33:49.989245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:49.989202 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" podStartSLOduration=1.989187207 podStartE2EDuration="1.989187207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:49.98342053 +0000 UTC m=+1256.387361483" watchObservedRunningTime="2026-11-03 22:33:49.989187207 +0000 UTC m=+1256.393128173" Nov 03 22:33:50.957920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:50.957886 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" event={"ID":"e7602f85-77a4-4ed5-991f-08ad57b49be3","Type":"ContainerStarted","Data":"52379da38d9b289285f718d2c86827b6aa00ee3b9e643839f6d2018e7edfc54e"} Nov 03 22:33:50.997283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:50.997233 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" podStartSLOduration=2.99721823 podStartE2EDuration="2.99721823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:50.99487365 +0000 UTC m=+1257.398814629" watchObservedRunningTime="2026-11-03 22:33:50.99721823 +0000 UTC m=+1257.401159205" Nov 03 22:33:51.342355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:51.342245 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:52.033941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:52.033909 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx"] Nov 03 22:33:52.045838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:52.045806 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv"] Nov 03 22:33:52.056456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:52.056420 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f20189f-vq4gx"] Nov 03 22:33:52.061986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:52.061949 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f9feb45-xgnwv"] Nov 03 22:33:53.737685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:53.737653 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe72668-863a-4438-8eb2-128814508bbe" path="/var/lib/kubelet/pods/bfe72668-863a-4438-8eb2-128814508bbe/volumes" Nov 03 22:33:53.738257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:53.738145 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0188a6-db35-49b7-a3ae-75747cf438cb" path="/var/lib/kubelet/pods/df0188a6-db35-49b7-a3ae-75747cf438cb/volumes" Nov 03 22:33:55.529258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.529223 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l"] Nov 03 22:33:55.536015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.535982 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.556829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.556776 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8"] Nov 03 22:33:55.562617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.562591 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:33:55.577667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.577636 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l"] Nov 03 22:33:55.597192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.597157 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8"] Nov 03 22:33:55.634098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.634059 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4c5345\"" Nov 03 22:33:55.635067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.634170 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-26f8ba-openshift-default-dockercfg-2tjmk\"" Nov 03 22:33:55.635067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.634394 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4c5345-dockercfg-4rpmw\"" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.663731 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-socket\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.663795 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-data\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.663867 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-token\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.664041 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c836105-532d-4c37-a4ca-520150b28f7d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8\" (UID: \"8c836105-532d-4c37-a4ca-520150b28f7d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.664096 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb0da7d-26b1-4802-b507-b552705b9d30-istiod-ca-cert\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.664172 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2475\" (UniqueName: \"kubernetes.io/projected/8c836105-532d-4c37-a4ca-520150b28f7d-kube-api-access-f2475\") pod \"payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8\" (UID: \"8c836105-532d-4c37-a4ca-520150b28f7d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.664222 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9v6\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-kube-api-access-2n9v6\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.664266 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-envoy\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.664340 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-credential-socket\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.664381 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-podinfo\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.673359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.664423 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-certs\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.795375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.774696 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-token\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.795831 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.795801 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c836105-532d-4c37-a4ca-520150b28f7d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8\" (UID: \"8c836105-532d-4c37-a4ca-520150b28f7d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:33:55.796018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.796002 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb0da7d-26b1-4802-b507-b552705b9d30-istiod-ca-cert\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.796180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.796166 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2475\" (UniqueName: \"kubernetes.io/projected/8c836105-532d-4c37-a4ca-520150b28f7d-kube-api-access-f2475\") pod \"payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8\" (UID: \"8c836105-532d-4c37-a4ca-520150b28f7d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:33:55.796337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.796317 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9v6\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-kube-api-access-2n9v6\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.796481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.796466 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-envoy\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.796616 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.796603 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-credential-socket\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.796739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.796723 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-podinfo\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.796883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.796867 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-certs\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.797162 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.797145 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-socket\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.797306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.797274 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-data\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.810316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.797912 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-data\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.827312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.803712 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-token\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.827312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.809720 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb0da7d-26b1-4802-b507-b552705b9d30-istiod-ca-cert\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.827312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.810475 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c836105-532d-4c37-a4ca-520150b28f7d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8\" (UID: \"8c836105-532d-4c37-a4ca-520150b28f7d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:33:55.827312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.813662 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-credential-socket\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.827312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.813924 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-certs\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.827312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.814186 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-socket\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.827312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.815905 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-envoy\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.827312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.818648 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-podinfo\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.848698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.839050 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9v6\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-kube-api-access-2n9v6\") pod \"e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.848698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.841023 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2475\" (UniqueName: \"kubernetes.io/projected/8c836105-532d-4c37-a4ca-520150b28f7d-kube-api-access-f2475\") pod \"payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8\" (UID: \"8c836105-532d-4c37-a4ca-520150b28f7d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:33:55.861862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.861828 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:55.901316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:55.899698 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:33:56.123266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.119885 4903 scope.go:122] "RemoveContainer" containerID="f416d9bcb90e2b3cc842d7b0ecb2ddfce18a1f3e496d15f815dbafc76aad00c2" Nov 03 22:33:56.161542 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.159897 4903 scope.go:122] "RemoveContainer" containerID="cf2ff261aa5fd5440d65cc4732bf8507d7786d1689e37c411692588a2c2441dd" Nov 03 22:33:56.177036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.177009 4903 scope.go:122] "RemoveContainer" containerID="1e1903b6b52917b5469cc015be3eb13df7a67441d7f4d80abdabe488f04bfe6b" Nov 03 22:33:56.236173 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:33:56.234966 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb0da7d_26b1_4802_b507_b552705b9d30.slice/crio-a560d9fab787a611ea5305fd2a79b56d410dad8e16781114053676d7bbe5ac6b WatchSource:0}: Error finding container a560d9fab787a611ea5305fd2a79b56d410dad8e16781114053676d7bbe5ac6b: Status 404 returned error can't find the container with id a560d9fab787a611ea5305fd2a79b56d410dad8e16781114053676d7bbe5ac6b Nov 03 22:33:56.239154 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.239109 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:33:56.239248 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.239203 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:33:56.239337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.239249 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:33:56.245808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.245784 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l"] Nov 03 22:33:56.260361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.260337 4903 scope.go:122] "RemoveContainer" containerID="bc9186995207c13390ed7b38fadb1ed66e02f1425477457188ef2c0cc4cc5517" Nov 03 22:33:56.299557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.299517 4903 scope.go:122] "RemoveContainer" containerID="6f9f94898efd1e7226406d2be2124472658bf9f010d35fd59b6e1ef44c70bad9" Nov 03 22:33:56.314688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.314667 4903 scope.go:122] "RemoveContainer" containerID="52c2065395850c2b3477d12c80b853e6b958f37489664e7c183ace529fb2626f" Nov 03 22:33:56.337780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.337760 4903 scope.go:122] "RemoveContainer" containerID="0e9c9371ec2ec2668d2494a079d2c37daf372103fd77363f5579dcf30a5a89f2" Nov 03 22:33:56.350720 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:33:56.350690 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c836105_532d_4c37_a4ca_520150b28f7d.slice/crio-8f101cbbafb5b3d54de92e6e6b12d768e9fdcc2899790241106b11c3928b7d30 WatchSource:0}: Error finding container 8f101cbbafb5b3d54de92e6e6b12d768e9fdcc2899790241106b11c3928b7d30: Status 404 returned error can't find the container with id 8f101cbbafb5b3d54de92e6e6b12d768e9fdcc2899790241106b11c3928b7d30 Nov 03 22:33:56.351963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.351938 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8"] Nov 03 22:33:56.356453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:56.356429 4903 scope.go:122] "RemoveContainer" containerID="0c3c85e5141f491f5152b687c4796faca344076f06c5ae3953fb21c60a19512c" Nov 03 22:33:57.032372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:57.032330 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" event={"ID":"2fb0da7d-26b1-4802-b507-b552705b9d30","Type":"ContainerStarted","Data":"cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100"} Nov 03 22:33:57.032893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:57.032401 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" event={"ID":"2fb0da7d-26b1-4802-b507-b552705b9d30","Type":"ContainerStarted","Data":"a560d9fab787a611ea5305fd2a79b56d410dad8e16781114053676d7bbe5ac6b"} Nov 03 22:33:57.035111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:57.034549 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" event={"ID":"8c836105-532d-4c37-a4ca-520150b28f7d","Type":"ContainerStarted","Data":"c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f"} Nov 03 22:33:57.035111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:57.034602 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" event={"ID":"8c836105-532d-4c37-a4ca-520150b28f7d","Type":"ContainerStarted","Data":"8f101cbbafb5b3d54de92e6e6b12d768e9fdcc2899790241106b11c3928b7d30"} Nov 03 22:33:57.035111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:57.034639 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:33:57.087059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:57.087006 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" podStartSLOduration=2.086991704 podStartE2EDuration="2.086991704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:57.080435736 +0000 UTC m=+1263.484376711" watchObservedRunningTime="2026-11-03 22:33:57.086991704 +0000 UTC m=+1263.490932746" Nov 03 22:33:57.122468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:57.122412 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" podStartSLOduration=2.122398655 podStartE2EDuration="2.122398655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:33:57.11689333 +0000 UTC m=+1263.520834307" watchObservedRunningTime="2026-11-03 22:33:57.122398655 +0000 UTC m=+1263.526339630" Nov 03 22:33:57.862004 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:57.861963 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:57.867816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:57.867789 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:58.044069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:58.044022 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:58.045110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:58.045087 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:33:58.056288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:58.056260 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs"] Nov 03 22:33:58.061046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:58.061025 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde9e295-rb9cs"] Nov 03 22:33:59.033036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:59.032999 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps"] Nov 03 22:33:59.042992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:59.042964 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8ff0633-qmlps"] Nov 03 22:33:59.342996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:59.342925 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:59.356184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:59.356163 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:33:59.740175 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:59.740134 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788ad52d-efde-4a24-bb83-63453db6e0a0" path="/var/lib/kubelet/pods/788ad52d-efde-4a24-bb83-63453db6e0a0/volumes" Nov 03 22:33:59.740996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:33:59.740971 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdb3ef8d-6902-45fb-a727-25c1737cf73f" path="/var/lib/kubelet/pods/bdb3ef8d-6902-45fb-a727-25c1737cf73f/volumes" Nov 03 22:34:00.075068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:00.074990 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:34:00.338209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:00.338130 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:34:00.962395 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:00.962363 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 22:34:01.980262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:01.980223 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl"] Nov 03 22:34:01.994738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:01.994699 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl"] Nov 03 22:34:01.994939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:01.994911 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:02.041091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.041056 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-26f8ba\"" Nov 03 22:34:02.041563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.041543 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-26f8ba-dockercfg-x9lc5\"" Nov 03 22:34:02.155179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.153714 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk"] Nov 03 22:34:02.167115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.167082 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:02.204226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.184347 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk"] Nov 03 22:34:02.204226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.196221 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdhk\" (UniqueName: \"kubernetes.io/projected/84afa961-a34b-44a0-9e02-68a6dc7a04dc-kube-api-access-tcdhk\") pod \"payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl\" (UID: \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:02.204226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.196269 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afa961-a34b-44a0-9e02-68a6dc7a04dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl\" (UID: \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:02.204226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.196367 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk\" (UID: \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:02.204226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.196517 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n88t\" (UniqueName: \"kubernetes.io/projected/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-kube-api-access-7n88t\") pod \"payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk\" (UID: \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:02.331905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.320428 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdhk\" (UniqueName: \"kubernetes.io/projected/84afa961-a34b-44a0-9e02-68a6dc7a04dc-kube-api-access-tcdhk\") pod \"payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl\" (UID: \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:02.331905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.320498 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afa961-a34b-44a0-9e02-68a6dc7a04dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl\" (UID: \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:02.331905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.320600 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk\" (UID: \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:02.331905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.320788 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n88t\" (UniqueName: \"kubernetes.io/projected/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-kube-api-access-7n88t\") pod \"payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk\" (UID: \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:02.352623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.347558 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afa961-a34b-44a0-9e02-68a6dc7a04dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl\" (UID: \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:02.352623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.348449 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk\" (UID: \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:02.352623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.351239 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n88t\" (UniqueName: \"kubernetes.io/projected/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-kube-api-access-7n88t\") pod \"payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk\" (UID: \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:02.389838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.389798 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:02.398223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.397113 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdhk\" (UniqueName: \"kubernetes.io/projected/84afa961-a34b-44a0-9e02-68a6dc7a04dc-kube-api-access-tcdhk\") pod \"payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl\" (UID: \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:02.685891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.685852 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:02.930797 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:34:02.930520 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2017c3ae_6dda_4b3f_bbb0_108c7b264bb1.slice/crio-715201807dd51dd9570d1e0b17570c40e992c709db54a3e82ff3a24e62f7c6aa WatchSource:0}: Error finding container 715201807dd51dd9570d1e0b17570c40e992c709db54a3e82ff3a24e62f7c6aa: Status 404 returned error can't find the container with id 715201807dd51dd9570d1e0b17570c40e992c709db54a3e82ff3a24e62f7c6aa Nov 03 22:34:02.938432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:02.937916 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk"] Nov 03 22:34:03.239349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:03.224231 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" event={"ID":"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1","Type":"ContainerStarted","Data":"715201807dd51dd9570d1e0b17570c40e992c709db54a3e82ff3a24e62f7c6aa"} Nov 03 22:34:03.901900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:03.901855 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl"] Nov 03 22:34:03.904976 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:34:03.904947 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84afa961_a34b_44a0_9e02_68a6dc7a04dc.slice/crio-d01f803632b55455c3e9795a379a96578b5b63ef450ed132afa585a8b38b4add WatchSource:0}: Error finding container d01f803632b55455c3e9795a379a96578b5b63ef450ed132afa585a8b38b4add: Status 404 returned error can't find the container with id d01f803632b55455c3e9795a379a96578b5b63ef450ed132afa585a8b38b4add Nov 03 22:34:04.235633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:04.235594 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" event={"ID":"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1","Type":"ContainerStarted","Data":"f8447395ad8bc12c7623e00e428ea7e9da1253f159c963350f981e757fc741bb"} Nov 03 22:34:04.237444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:04.237410 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" event={"ID":"84afa961-a34b-44a0-9e02-68a6dc7a04dc","Type":"ContainerStarted","Data":"1d0b761642afb5ee2225bc6bf28f11373c431eef793bee72bcf5965c72fed8f3"} Nov 03 22:34:04.237584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:04.237471 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" event={"ID":"84afa961-a34b-44a0-9e02-68a6dc7a04dc","Type":"ContainerStarted","Data":"d01f803632b55455c3e9795a379a96578b5b63ef450ed132afa585a8b38b4add"} Nov 03 22:34:04.237584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:04.237502 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:04.238439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:04.238419 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:04.328199 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:04.328148 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" podStartSLOduration=3.328133658 podStartE2EDuration="3.328133658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:04.324554331 +0000 UTC m=+1270.728495306" watchObservedRunningTime="2026-11-03 22:34:04.328133658 +0000 UTC m=+1270.732074634" Nov 03 22:34:04.398518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:04.398344 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" podStartSLOduration=3.398320876 podStartE2EDuration="3.398320876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:04.364767495 +0000 UTC m=+1270.768708487" watchObservedRunningTime="2026-11-03 22:34:04.398320876 +0000 UTC m=+1270.802261852" Nov 03 22:34:08.086317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:08.085836 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:34:11.028842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:11.028806 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt"] Nov 03 22:34:11.041234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:11.041200 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2478f748-mhwkt"] Nov 03 22:34:11.742180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:11.742138 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00605c20-be93-44bc-8e74-c80d43371bf2" path="/var/lib/kubelet/pods/00605c20-be93-44bc-8e74-c80d43371bf2/volumes" Nov 03 22:34:15.299548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:15.299517 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 22:34:16.317031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.317002 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 22:34:16.531521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.531489 4903 generic.go:356] "Generic (PLEG): container finished" podID="f335df24-eab6-49f0-a307-86fea13f9402" containerID="58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a" exitCode=0 Nov 03 22:34:16.531726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.531562 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" event={"ID":"f335df24-eab6-49f0-a307-86fea13f9402","Type":"ContainerDied","Data":"58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a"} Nov 03 22:34:16.532186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.532161 4903 scope.go:122] "RemoveContainer" containerID="58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a" Nov 03 22:34:16.534025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.533991 4903 generic.go:356] "Generic (PLEG): container finished" podID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerID="997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d" exitCode=0 Nov 03 22:34:16.534129 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.534042 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" event={"ID":"f9847228-3a90-4625-a6b0-e17b44bed10e","Type":"ContainerDied","Data":"997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d"} Nov 03 22:34:16.534424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.534391 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:34:16.534531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.534485 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:34:16.534531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.534508 4903 scope.go:122] "RemoveContainer" containerID="997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d" Nov 03 22:34:16.534531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.534526 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:34:16.536429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.536396 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:34:16.536511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.536477 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:34:16.536550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:16.536541 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:34:17.045645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:17.045608 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9"] Nov 03 22:34:17.053601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:17.053556 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b48b994d-qztv9"] Nov 03 22:34:17.588318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:17.588184 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" event={"ID":"f335df24-eab6-49f0-a307-86fea13f9402","Type":"ContainerStarted","Data":"83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e"} Nov 03 22:34:17.593809 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:17.593768 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" event={"ID":"f9847228-3a90-4625-a6b0-e17b44bed10e","Type":"ContainerStarted","Data":"a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46"} Nov 03 22:34:17.731141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:17.731115 4903 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:34:17.731431 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:17.731383 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:34:17.738882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:17.738850 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b" path="/var/lib/kubelet/pods/7b9c8f5b-aeec-48b8-8fd2-6941410ccf4b/volumes" Nov 03 22:34:17.739480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:17.739459 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:34:18.224614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:18.224582 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:34:18.389323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:18.388740 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:34:18.392225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:18.392194 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:34:18.392830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:18.392357 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:34:18.450593 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:34:18.450550 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:34:18.730935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:18.730900 4903 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:34:18.731361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:18.730954 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:34:19.391320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:19.388673 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:34:19.391320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:19.388735 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:34:19.730328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:19.730276 4903 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:34:19.730531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:19.730353 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:34:20.255435 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:20.255395 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:34:20.385718 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:20.385682 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:34:20.386000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:20.385970 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:34:20.729766 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:20.729729 4903 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:34:20.730014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:20.729773 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:34:21.391375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:21.387686 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:34:21.391375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:21.387748 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:34:21.729930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:21.729860 4903 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:34:21.730138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:21.729953 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:34:22.389716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:22.389676 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 22:34:22.390040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:22.390006 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 22:34:22.729825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:22.729753 4903 patch_prober.go:29] interesting pod/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 22:34:22.730420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:22.730397 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 22:34:23.052466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.052372 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8"] Nov 03 22:34:23.064338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.064288 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz"] Nov 03 22:34:23.074382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.074345 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a652e171-jmst8"] Nov 03 22:34:23.081555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.081506 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b7f06-qh2zz"] Nov 03 22:34:23.391060 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.390969 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:34:23.392258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.392235 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 22:34:23.753761 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.753717 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55127199-7e7c-4b3a-b500-c344cc2d61c7" path="/var/lib/kubelet/pods/55127199-7e7c-4b3a-b500-c344cc2d61c7/volumes" Nov 03 22:34:23.754458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.754413 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a06f81-9ca7-4966-9b90-c940be6aa899" path="/var/lib/kubelet/pods/d5a06f81-9ca7-4966-9b90-c940be6aa899/volumes" Nov 03 22:34:23.756060 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.756039 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:34:23.758888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:23.758868 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:34:29.880516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:29.880480 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4"] Nov 03 22:34:29.880932 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:29.880691 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" containerID="cri-o://753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63" gracePeriod=30 Nov 03 22:34:35.188049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.188020 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:34:35.217937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.217906 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d74942a-efdf-4726-a427-efd8c125ec7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74942a-efdf-4726-a427-efd8c125ec7e-istiod-ca-cert\") pod \"0d74942a-efdf-4726-a427-efd8c125ec7e\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " Nov 03 22:34:35.218119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.217984 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-token\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-token\") pod \"0d74942a-efdf-4726-a427-efd8c125ec7e\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " Nov 03 22:34:35.218119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218081 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-podinfo\") pod \"0d74942a-efdf-4726-a427-efd8c125ec7e\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " Nov 03 22:34:35.218238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218206 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-kube-api-access-hxvzb\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-kube-api-access-hxvzb\") pod \"0d74942a-efdf-4726-a427-efd8c125ec7e\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " Nov 03 22:34:35.218311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218275 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-socket\") pod \"0d74942a-efdf-4726-a427-efd8c125ec7e\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " Nov 03 22:34:35.218381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218368 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-credential-socket\") pod \"0d74942a-efdf-4726-a427-efd8c125ec7e\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " Nov 03 22:34:35.218445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218390 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d74942a-efdf-4726-a427-efd8c125ec7e-istiod-ca-cert" pod "0d74942a-efdf-4726-a427-efd8c125ec7e" (UID: "0d74942a-efdf-4726-a427-efd8c125ec7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:34:35.218445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218408 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-certs\") pod \"0d74942a-efdf-4726-a427-efd8c125ec7e\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " Nov 03 22:34:35.218550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218463 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-data\") pod \"0d74942a-efdf-4726-a427-efd8c125ec7e\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " Nov 03 22:34:35.218613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218604 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-envoy\") pod \"0d74942a-efdf-4726-a427-efd8c125ec7e\" (UID: \"0d74942a-efdf-4726-a427-efd8c125ec7e\") " Nov 03 22:34:35.218714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218696 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-credential-socket" pod "0d74942a-efdf-4726-a427-efd8c125ec7e" (UID: "0d74942a-efdf-4726-a427-efd8c125ec7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:35.218918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218894 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-certs" pod "0d74942a-efdf-4726-a427-efd8c125ec7e" (UID: "0d74942a-efdf-4726-a427-efd8c125ec7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:35.218956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218906 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-data" pod "0d74942a-efdf-4726-a427-efd8c125ec7e" (UID: "0d74942a-efdf-4726-a427-efd8c125ec7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:35.218956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.218833 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-socket" pod "0d74942a-efdf-4726-a427-efd8c125ec7e" (UID: "0d74942a-efdf-4726-a427-efd8c125ec7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:35.219571 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.219544 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d74942a-efdf-4726-a427-efd8c125ec7e-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:35.219680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.219575 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:35.219680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.219590 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:35.219680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.219605 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:35.219680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.219620 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:35.221016 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.220989 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-token" pod "0d74942a-efdf-4726-a427-efd8c125ec7e" (UID: "0d74942a-efdf-4726-a427-efd8c125ec7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:35.221125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.221042 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-envoy" pod "0d74942a-efdf-4726-a427-efd8c125ec7e" (UID: "0d74942a-efdf-4726-a427-efd8c125ec7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:34:35.221279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.221265 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-podinfo" pod "0d74942a-efdf-4726-a427-efd8c125ec7e" (UID: "0d74942a-efdf-4726-a427-efd8c125ec7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:34:35.221445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.221429 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-kube-api-access-hxvzb" pod "0d74942a-efdf-4726-a427-efd8c125ec7e" (UID: "0d74942a-efdf-4726-a427-efd8c125ec7e"). InnerVolumeSpecName "kube-api-access-hxvzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:35.321225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.321192 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:35.321225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.321221 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxvzb\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-kube-api-access-hxvzb\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:35.321225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.321231 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:35.321488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.321240 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d74942a-efdf-4726-a427-efd8c125ec7e-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:35.843240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.843208 4903 generic.go:356] "Generic (PLEG): container finished" podID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerID="753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63" exitCode=0 Nov 03 22:34:35.843465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.843255 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" Nov 03 22:34:35.843465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.843309 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" event={"ID":"0d74942a-efdf-4726-a427-efd8c125ec7e","Type":"ContainerDied","Data":"753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63"} Nov 03 22:34:35.843465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.843371 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4" event={"ID":"0d74942a-efdf-4726-a427-efd8c125ec7e","Type":"ContainerDied","Data":"fe967024fe01f45a541b39b56bbaceed11d3fe5fe0502e347fb7c1cf356e03a4"} Nov 03 22:34:35.843465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.843396 4903 scope.go:122] "RemoveContainer" containerID="753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63" Nov 03 22:34:35.855149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.855133 4903 scope.go:122] "RemoveContainer" containerID="753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63" Nov 03 22:34:35.855478 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:34:35.855455 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63\": container with ID starting with 753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63 not found: ID does not exist" containerID="753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63" Nov 03 22:34:35.855563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.855487 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63"} err="failed to get container status \"753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63\": rpc error: code = NotFound desc = could not find container \"753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63\": container with ID starting with 753d460c6b069a44bab15a0401840e936aaa7bca64bae8a4f3137d2267864b63 not found: ID does not exist" Nov 03 22:34:35.902701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.902668 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4"] Nov 03 22:34:35.914996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:35.914961 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-jrmj4"] Nov 03 22:34:36.037942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.037808 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2"] Nov 03 22:34:36.045757 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.045714 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9515d5c-fmvf2"] Nov 03 22:34:36.496318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.495278 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll"] Nov 03 22:34:36.496878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.496637 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" Nov 03 22:34:36.496878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.496661 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" Nov 03 22:34:36.497119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.496957 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" containerName="istio-proxy" Nov 03 22:34:36.604316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.591546 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll"] Nov 03 22:34:36.604316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.591689 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.606177 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.606103 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:34:36.631340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.610627 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:34:36.631340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.610802 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:34:36.631340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.611162 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:34:36.664825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.664785 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baeaa733-ldbll\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.665259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.665232 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baeaa733-ldbll\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.665485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.665467 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plgkk\" (UniqueName: \"kubernetes.io/projected/2657aefc-f8ec-4dfa-8762-bfbc6322542c-kube-api-access-plgkk\") pod \"maas-api-revoke-sub-baeaa733-ldbll\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.767141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.767050 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baeaa733-ldbll\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.767341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.767154 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plgkk\" (UniqueName: \"kubernetes.io/projected/2657aefc-f8ec-4dfa-8762-bfbc6322542c-kube-api-access-plgkk\") pod \"maas-api-revoke-sub-baeaa733-ldbll\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.767483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.767459 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baeaa733-ldbll\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.768633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.768606 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baeaa733-ldbll\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.771595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.771569 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baeaa733-ldbll\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.790231 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.790054 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plgkk\" (UniqueName: \"kubernetes.io/projected/2657aefc-f8ec-4dfa-8762-bfbc6322542c-kube-api-access-plgkk\") pod \"maas-api-revoke-sub-baeaa733-ldbll\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:36.916856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:36.916825 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:37.309902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:37.309867 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll"] Nov 03 22:34:38.075871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:38.075821 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d74942a-efdf-4726-a427-efd8c125ec7e" path="/var/lib/kubelet/pods/0d74942a-efdf-4726-a427-efd8c125ec7e/volumes" Nov 03 22:34:38.076599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:38.076576 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb38697-525b-436f-bca0-ea1e391b112d" path="/var/lib/kubelet/pods/4cb38697-525b-436f-bca0-ea1e391b112d/volumes" Nov 03 22:34:38.077016 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:38.076994 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" event={"ID":"2657aefc-f8ec-4dfa-8762-bfbc6322542c","Type":"ContainerStarted","Data":"79be0a22e287e598b902cbfddbbfcc00d5360048d7ded073e1e00ce570d2430a"} Nov 03 22:34:39.076542 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:39.076506 4903 generic.go:356] "Generic (PLEG): container finished" podID="2657aefc-f8ec-4dfa-8762-bfbc6322542c" containerID="16923e7522c0c398a59b619412c1f15ed50670b5ed762eb5dad1d49bafc12d45" exitCode=0 Nov 03 22:34:39.076934 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:39.076597 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" event={"ID":"2657aefc-f8ec-4dfa-8762-bfbc6322542c","Type":"ContainerDied","Data":"16923e7522c0c398a59b619412c1f15ed50670b5ed762eb5dad1d49bafc12d45"} Nov 03 22:34:40.314399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.314366 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:40.384165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.384071 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-xks-ca\") pod \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " Nov 03 22:34:40.384165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.384136 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2657aefc-f8ec-4dfa-8762-bfbc6322542c-kube-api-access-plgkk\" (UniqueName: \"kubernetes.io/projected/2657aefc-f8ec-4dfa-8762-bfbc6322542c-kube-api-access-plgkk\") pod \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " Nov 03 22:34:40.384493 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.384311 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-openshift-ca\") pod \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\" (UID: \"2657aefc-f8ec-4dfa-8762-bfbc6322542c\") " Nov 03 22:34:40.384936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.384859 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-openshift-ca" pod "2657aefc-f8ec-4dfa-8762-bfbc6322542c" (UID: "2657aefc-f8ec-4dfa-8762-bfbc6322542c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:34:40.385233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.385195 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:40.386665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.386626 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2657aefc-f8ec-4dfa-8762-bfbc6322542c-kube-api-access-plgkk" pod "2657aefc-f8ec-4dfa-8762-bfbc6322542c" (UID: "2657aefc-f8ec-4dfa-8762-bfbc6322542c"). InnerVolumeSpecName "kube-api-access-plgkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:34:40.387286 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.387259 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-xks-ca" pod "2657aefc-f8ec-4dfa-8762-bfbc6322542c" (UID: "2657aefc-f8ec-4dfa-8762-bfbc6322542c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:34:40.488144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.488111 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2657aefc-f8ec-4dfa-8762-bfbc6322542c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:40.488144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:40.488138 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plgkk\" (UniqueName: \"kubernetes.io/projected/2657aefc-f8ec-4dfa-8762-bfbc6322542c-kube-api-access-plgkk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:34:41.101708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:41.101678 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" Nov 03 22:34:41.101883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:41.101676 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll" event={"ID":"2657aefc-f8ec-4dfa-8762-bfbc6322542c","Type":"ContainerDied","Data":"79be0a22e287e598b902cbfddbbfcc00d5360048d7ded073e1e00ce570d2430a"} Nov 03 22:34:41.101883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:41.101825 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79be0a22e287e598b902cbfddbbfcc00d5360048d7ded073e1e00ce570d2430a" Nov 03 22:34:48.385800 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:34:48.385754 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:34:56.091603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.091566 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq"] Nov 03 22:34:56.092427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.092412 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2657aefc-f8ec-4dfa-8762-bfbc6322542c" containerName="revoke-keys" Nov 03 22:34:56.092427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.092427 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2657aefc-f8ec-4dfa-8762-bfbc6322542c" containerName="revoke-keys" Nov 03 22:34:56.092598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.092586 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2657aefc-f8ec-4dfa-8762-bfbc6322542c" containerName="revoke-keys" Nov 03 22:34:56.095791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.095774 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:34:56.139143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.139113 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-32fdf2-dockercfg-rzln2\"" Nov 03 22:34:56.142654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.139486 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-32fdf2\"" Nov 03 22:34:56.147885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.147854 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq"] Nov 03 22:34:56.152639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.152615 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6"] Nov 03 22:34:56.156719 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.156699 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:34:56.173278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.173243 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6"] Nov 03 22:34:56.280765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.280734 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d41e8-3176-4768-84ff-2aae8a277e5f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6\" (UID: \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\") " pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:34:56.281028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.281005 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm6pf\" (UniqueName: \"kubernetes.io/projected/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-kube-api-access-nm6pf\") pod \"payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq\" (UID: \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:34:56.281127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.281096 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57z4l\" (UniqueName: \"kubernetes.io/projected/0d4d41e8-3176-4768-84ff-2aae8a277e5f-kube-api-access-57z4l\") pod \"payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6\" (UID: \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\") " pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:34:56.281253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.281237 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq\" (UID: \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:34:56.383165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.383079 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d41e8-3176-4768-84ff-2aae8a277e5f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6\" (UID: \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\") " pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:34:56.383165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.383119 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm6pf\" (UniqueName: \"kubernetes.io/projected/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-kube-api-access-nm6pf\") pod \"payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq\" (UID: \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:34:56.383165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.383161 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57z4l\" (UniqueName: \"kubernetes.io/projected/0d4d41e8-3176-4768-84ff-2aae8a277e5f-kube-api-access-57z4l\") pod \"payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6\" (UID: \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\") " pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:34:56.383514 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.383230 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq\" (UID: \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:34:56.384069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.384045 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq\" (UID: \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:34:56.384116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.384065 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d41e8-3176-4768-84ff-2aae8a277e5f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6\" (UID: \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\") " pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:34:56.410275 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.410246 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57z4l\" (UniqueName: \"kubernetes.io/projected/0d4d41e8-3176-4768-84ff-2aae8a277e5f-kube-api-access-57z4l\") pod \"payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6\" (UID: \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\") " pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:34:56.410433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.410414 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm6pf\" (UniqueName: \"kubernetes.io/projected/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-kube-api-access-nm6pf\") pod \"payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq\" (UID: \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:34:56.478044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.478006 4903 scope.go:122] "RemoveContainer" containerID="ceebb67c3c5dda4c494cf85ca6f6ec0923810c8ae2bbfbe982616794dabb4b43" Nov 03 22:34:56.491724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.491696 4903 scope.go:122] "RemoveContainer" containerID="848133600f43db0b5f68cc2cc5419ad21d2eea214a831702e9a634aeab39f644" Nov 03 22:34:56.503169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.503143 4903 scope.go:122] "RemoveContainer" containerID="2cd878d2acf2ecf1b1174b855be91c1f972d7cff46df1bb6f1f8360050526980" Nov 03 22:34:56.516498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.516472 4903 scope.go:122] "RemoveContainer" containerID="022f79b5250c9f870808a19aa86f0156cce083242453f7cad9a5843d115830de" Nov 03 22:34:56.531235 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.531215 4903 scope.go:122] "RemoveContainer" containerID="597950f6499dbf4c0d259ef189d05b789d4b5f84594322301c2b41044af00fb6" Nov 03 22:34:56.546634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.546613 4903 scope.go:122] "RemoveContainer" containerID="f0d28b5b5cd39021ab83c964195178f03f3401a5ec76fd3bacba255b5cd0e1a1" Nov 03 22:34:56.560588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.560563 4903 scope.go:122] "RemoveContainer" containerID="ac2213b8db97a79f4955b4e70c02765947d47f4ed2bccf00987302214c187e8b" Nov 03 22:34:56.574679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.574656 4903 scope.go:122] "RemoveContainer" containerID="c690009d05d9879ea3f473199fc32bd0078a91bfd78ab9b8c9dfa5217996e323" Nov 03 22:34:56.594255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.594223 4903 scope.go:122] "RemoveContainer" containerID="8671cfd38dd76227dee8a76edb71ba377e454196fa3d80594a58e294a99111dc" Nov 03 22:34:56.594515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.594456 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:34:56.619795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.619768 4903 scope.go:122] "RemoveContainer" containerID="021cae792cf493dc62e6a222f5fe79a28d494c1f7ea2c2615f275456e2bfa9b4" Nov 03 22:34:56.704365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.700596 4903 scope.go:122] "RemoveContainer" containerID="6fa735968c401c70dff0e55c850deb2f149f520e32001e4bf9923edc43a0bd7f" Nov 03 22:34:56.726899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.711469 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:34:56.772262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.772234 4903 scope.go:122] "RemoveContainer" containerID="4f8e2b1bd6525e78239e4a69c4b4abd047b6a2dd2ea1f55183e267f734f3ad19" Nov 03 22:34:56.874513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.873907 4903 scope.go:122] "RemoveContainer" containerID="7553b2b341ffbca8b96e5900d0c5109a73cf140541726e436f16ea6fb8936409" Nov 03 22:34:56.908303 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.908265 4903 scope.go:122] "RemoveContainer" containerID="8c5826d5a18225879fc7f89b3ab220db12014a7d17ecda25c9f5991086604b46" Nov 03 22:34:56.926026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.925995 4903 scope.go:122] "RemoveContainer" containerID="12f4f7cd63bc9f332a2a235c1c36b8dc06f923da61c0eec2c6b1f949fd8f3df9" Nov 03 22:34:56.939553 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.939530 4903 scope.go:122] "RemoveContainer" containerID="b18388d1f6f9e526595d860bc028fcbb22f283fa457e2e56903c9fc61c739063" Nov 03 22:34:56.954186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:56.954166 4903 scope.go:122] "RemoveContainer" containerID="84c0aaf1b4f88b1d9bf19f9f0b893f24de1bed49707492cac3316592e014d151" Nov 03 22:34:57.044870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.044834 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6"] Nov 03 22:34:57.099963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.099935 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq"] Nov 03 22:34:57.104429 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:34:57.104399 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fc63a8_8b2a_4eca_9d24_74e727a8e3c9.slice/crio-bccbd2fec40e5d657e766414979e99fa877fbbeac4ad505229525bdf23e3a7df WatchSource:0}: Error finding container bccbd2fec40e5d657e766414979e99fa877fbbeac4ad505229525bdf23e3a7df: Status 404 returned error can't find the container with id bccbd2fec40e5d657e766414979e99fa877fbbeac4ad505229525bdf23e3a7df Nov 03 22:34:57.428438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.428392 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" event={"ID":"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9","Type":"ContainerStarted","Data":"886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93"} Nov 03 22:34:57.428630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.428495 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" event={"ID":"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9","Type":"ContainerStarted","Data":"bccbd2fec40e5d657e766414979e99fa877fbbeac4ad505229525bdf23e3a7df"} Nov 03 22:34:57.428702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.428656 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:34:57.430288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.430260 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" event={"ID":"0d4d41e8-3176-4768-84ff-2aae8a277e5f","Type":"ContainerStarted","Data":"f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283"} Nov 03 22:34:57.430450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.430353 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" event={"ID":"0d4d41e8-3176-4768-84ff-2aae8a277e5f","Type":"ContainerStarted","Data":"dfb1c2ba9d5ed72e683d5a1b56891ddf64650b2f88b21c8a6cfa43a5066c491f"} Nov 03 22:34:57.430450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.430380 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:34:57.466233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.466180 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" podStartSLOduration=1.466166311 podStartE2EDuration="1.466166311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:57.465444422 +0000 UTC m=+1323.869385400" watchObservedRunningTime="2026-11-03 22:34:57.466166311 +0000 UTC m=+1323.870107320" Nov 03 22:34:57.528072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:34:57.527974 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" podStartSLOduration=1.527959176 podStartE2EDuration="1.527959176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:34:57.519543764 +0000 UTC m=+1323.923484739" watchObservedRunningTime="2026-11-03 22:34:57.527959176 +0000 UTC m=+1323.931900151" Nov 03 22:35:03.424690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.424647 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx"] Nov 03 22:35:03.430648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.430619 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:03.547322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.531967 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx"] Nov 03 22:35:03.560315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.547556 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bbfc2e\"" Nov 03 22:35:03.560315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.549191 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bbfc2e-dockercfg-x6wkj\"" Nov 03 22:35:03.629100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.629068 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bljj\" (UniqueName: \"kubernetes.io/projected/91626678-bacb-47cd-bc41-69ebdea13346-kube-api-access-8bljj\") pod \"payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx\" (UID: \"91626678-bacb-47cd-bc41-69ebdea13346\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:03.629451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.629428 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91626678-bacb-47cd-bc41-69ebdea13346-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx\" (UID: \"91626678-bacb-47cd-bc41-69ebdea13346\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:03.731043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.730931 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91626678-bacb-47cd-bc41-69ebdea13346-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx\" (UID: \"91626678-bacb-47cd-bc41-69ebdea13346\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:03.731531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.731509 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bljj\" (UniqueName: \"kubernetes.io/projected/91626678-bacb-47cd-bc41-69ebdea13346-kube-api-access-8bljj\") pod \"payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx\" (UID: \"91626678-bacb-47cd-bc41-69ebdea13346\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:03.732690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.732658 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91626678-bacb-47cd-bc41-69ebdea13346-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx\" (UID: \"91626678-bacb-47cd-bc41-69ebdea13346\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:03.753799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.753764 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bljj\" (UniqueName: \"kubernetes.io/projected/91626678-bacb-47cd-bc41-69ebdea13346-kube-api-access-8bljj\") pod \"payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx\" (UID: \"91626678-bacb-47cd-bc41-69ebdea13346\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:03.889319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:03.889132 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:04.219232 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:35:04.219193 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91626678_bacb_47cd_bc41_69ebdea13346.slice/crio-2a72c0c3539e61695d29982b3365429521a339e99beb5d6b1a1d639b696a4193 WatchSource:0}: Error finding container 2a72c0c3539e61695d29982b3365429521a339e99beb5d6b1a1d639b696a4193: Status 404 returned error can't find the container with id 2a72c0c3539e61695d29982b3365429521a339e99beb5d6b1a1d639b696a4193 Nov 03 22:35:04.220212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:04.220181 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx"] Nov 03 22:35:04.625582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:04.625481 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" event={"ID":"91626678-bacb-47cd-bc41-69ebdea13346","Type":"ContainerStarted","Data":"61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9"} Nov 03 22:35:04.625582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:04.625551 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" event={"ID":"91626678-bacb-47cd-bc41-69ebdea13346","Type":"ContainerStarted","Data":"2a72c0c3539e61695d29982b3365429521a339e99beb5d6b1a1d639b696a4193"} Nov 03 22:35:04.626105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:04.625593 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:04.667402 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:04.667350 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" podStartSLOduration=1.667332986 podStartE2EDuration="1.667332986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:04.661964651 +0000 UTC m=+1331.065905626" watchObservedRunningTime="2026-11-03 22:35:04.667332986 +0000 UTC m=+1331.071273960" Nov 03 22:35:06.825245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:06.825208 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:35:08.448643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:08.448613 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:35:08.452221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:08.452196 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:35:11.052344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.051425 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.065920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.065872 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" podUID="e7602f85-77a4-4ed5-991f-08ad57b49be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.106524 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.106460 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.122757 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.122697 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.247319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.237469 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.247319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.237773 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.247319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.238117 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 22:35:11.290384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.282992 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.290384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.283458 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 03 22:35:11.429166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.428426 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.471319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.456086 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.543792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.541533 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.543792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.541595 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:11.629012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.628962 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 22:35:11.755615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:11.755489 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 22:35:12.009373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:12.004861 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:12.077319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:12.075314 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" podUID="e7602f85-77a4-4ed5-991f-08ad57b49be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.169:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:35:12.077319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:12.076063 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:35:12.103718 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:12.103688 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:35:15.473936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:15.473890 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84"] Nov 03 22:35:15.474546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:15.474155 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" containerID="cri-o://ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334" gracePeriod=30 Nov 03 22:35:15.489725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:15.489691 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2"] Nov 03 22:35:15.489965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:15.489940 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" containerID="cri-o://a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb" gracePeriod=30 Nov 03 22:35:15.638779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:15.638683 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:35:15.843639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:15.843592 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26"] Nov 03 22:35:15.843844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:15.843815 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" containerID="cri-o://f9fef8b2e5916de7aa705ffb82a2cecbab2e546f0779ff74d430a96bb80baa89" gracePeriod=30 Nov 03 22:35:15.849790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:15.849762 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh"] Nov 03 22:35:15.850041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:15.850017 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" containerID="cri-o://02331e20d94cda190bdf624b4bd40bf0be04909881b04b3f1947908389166c11" gracePeriod=30 Nov 03 22:35:18.387900 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:18.387854 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:35:20.134647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.134615 4903 generic.go:356] "Generic (PLEG): container finished" podID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerID="cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100" exitCode=0 Nov 03 22:35:20.135099 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.134688 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" event={"ID":"2fb0da7d-26b1-4802-b507-b552705b9d30","Type":"ContainerDied","Data":"cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100"} Nov 03 22:35:20.135234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.135217 4903 scope.go:122] "RemoveContainer" containerID="cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100" Nov 03 22:35:20.138263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.138222 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:35:20.138452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.138396 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:35:20.138515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.138484 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:35:20.868581 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.868156 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:35:20.995393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.963055 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-certs\") pod \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " Nov 03 22:35:20.995393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.963155 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-podinfo\") pod \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " Nov 03 22:35:20.995393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.963231 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istiod-ca-cert\") pod \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " Nov 03 22:35:20.995393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.963324 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-envoy\") pod \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " Nov 03 22:35:20.995393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.963420 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-token\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-token\") pod \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " Nov 03 22:35:20.995393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.963531 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-data\") pod \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " Nov 03 22:35:20.995393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.963698 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-kube-api-access-xdmmn\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-kube-api-access-xdmmn\") pod \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " Nov 03 22:35:20.995393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.963802 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-socket\") pod \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " Nov 03 22:35:20.995393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.963878 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-credential-socket\") pod \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\" (UID: \"8ca6fd32-0245-4ea1-ae2c-81512aa4e979\") " Nov 03 22:35:20.996337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.996151 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-credential-socket" pod "8ca6fd32-0245-4ea1-ae2c-81512aa4e979" (UID: "8ca6fd32-0245-4ea1-ae2c-81512aa4e979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:20.997077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.997051 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-data" pod "8ca6fd32-0245-4ea1-ae2c-81512aa4e979" (UID: "8ca6fd32-0245-4ea1-ae2c-81512aa4e979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:20.997471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.997451 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-certs" pod "8ca6fd32-0245-4ea1-ae2c-81512aa4e979" (UID: "8ca6fd32-0245-4ea1-ae2c-81512aa4e979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:20.997871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.997848 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istiod-ca-cert" pod "8ca6fd32-0245-4ea1-ae2c-81512aa4e979" (UID: "8ca6fd32-0245-4ea1-ae2c-81512aa4e979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:20.998629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:20.998605 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-socket" pod "8ca6fd32-0245-4ea1-ae2c-81512aa4e979" (UID: "8ca6fd32-0245-4ea1-ae2c-81512aa4e979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.002947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.002912 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-kube-api-access-xdmmn" pod "8ca6fd32-0245-4ea1-ae2c-81512aa4e979" (UID: "8ca6fd32-0245-4ea1-ae2c-81512aa4e979"). InnerVolumeSpecName "kube-api-access-xdmmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.015337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.004405 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-podinfo" pod "8ca6fd32-0245-4ea1-ae2c-81512aa4e979" (UID: "8ca6fd32-0245-4ea1-ae2c-81512aa4e979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:21.015337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.004788 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-envoy" pod "8ca6fd32-0245-4ea1-ae2c-81512aa4e979" (UID: "8ca6fd32-0245-4ea1-ae2c-81512aa4e979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.015337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.013572 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:35:21.015337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.014834 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-token" pod "8ca6fd32-0245-4ea1-ae2c-81512aa4e979" (UID: "8ca6fd32-0245-4ea1-ae2c-81512aa4e979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.098208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.093769 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.098208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.093806 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.098208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.093821 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.098208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.093834 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.098208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.093848 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.098208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.093861 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.098208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.093874 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.098208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.093887 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.098208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.093900 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdmmn\" (UniqueName: \"kubernetes.io/projected/8ca6fd32-0245-4ea1-ae2c-81512aa4e979-kube-api-access-xdmmn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.202078 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.202042 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-envoy\") pod \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " Nov 03 22:35:21.202699 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.202674 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-token\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-token\") pod \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " Nov 03 22:35:21.203344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.203324 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istiod-ca-cert\") pod \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " Nov 03 22:35:21.203594 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.203565 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-podinfo\") pod \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " Nov 03 22:35:21.204067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.204046 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-socket\") pod \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " Nov 03 22:35:21.204400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.204383 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-kube-api-access-z6tw2\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-kube-api-access-z6tw2\") pod \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " Nov 03 22:35:21.204803 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.204784 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-certs\") pod \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " Nov 03 22:35:21.204997 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.204976 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-data\") pod \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " Nov 03 22:35:21.205164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.205140 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-credential-socket\") pod \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\" (UID: \"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e\") " Nov 03 22:35:21.206678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.206649 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-credential-socket" pod "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" (UID: "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.213712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.208811 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-data" pod "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" (UID: "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.213712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.209131 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istiod-ca-cert" pod "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" (UID: "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:21.213712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.210285 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-socket" pod "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" (UID: "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.213712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.210435 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-certs" pod "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" (UID: "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.218329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.217166 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-envoy" pod "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" (UID: "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.220675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.220613 4903 generic.go:356] "Generic (PLEG): container finished" podID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerID="02331e20d94cda190bdf624b4bd40bf0be04909881b04b3f1947908389166c11" exitCode=0 Nov 03 22:35:21.220880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.220800 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" event={"ID":"621fc439-b1b7-4770-bd4e-529b1fcffe22","Type":"ContainerDied","Data":"02331e20d94cda190bdf624b4bd40bf0be04909881b04b3f1947908389166c11"} Nov 03 22:35:21.227243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.224428 4903 generic.go:356] "Generic (PLEG): container finished" podID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerID="a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb" exitCode=0 Nov 03 22:35:21.227243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.224533 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" event={"ID":"8ca6fd32-0245-4ea1-ae2c-81512aa4e979","Type":"ContainerDied","Data":"a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb"} Nov 03 22:35:21.227243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.224580 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" event={"ID":"8ca6fd32-0245-4ea1-ae2c-81512aa4e979","Type":"ContainerDied","Data":"e74902fd1e30571a172b1a8256ceca4d4e591335ac199e89ae5c6e337c42013d"} Nov 03 22:35:21.227243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.224607 4903 scope.go:122] "RemoveContainer" containerID="a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb" Nov 03 22:35:21.227243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.224783 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2" Nov 03 22:35:21.230864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.229242 4903 generic.go:356] "Generic (PLEG): container finished" podID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerID="ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334" exitCode=0 Nov 03 22:35:21.230864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.229424 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" event={"ID":"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e","Type":"ContainerDied","Data":"ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334"} Nov 03 22:35:21.230864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.229471 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" event={"ID":"d0f7c6c6-3025-4adc-9695-0fbfb2ba017e","Type":"ContainerDied","Data":"8492cf1101ef918ebb52f2d652250c71e6b00a295579b05fb5e9765402ac0203"} Nov 03 22:35:21.230864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.229551 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84" Nov 03 22:35:21.236385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.232221 4903 generic.go:356] "Generic (PLEG): container finished" podID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerID="f9fef8b2e5916de7aa705ffb82a2cecbab2e546f0779ff74d430a96bb80baa89" exitCode=0 Nov 03 22:35:21.236385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.232357 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" event={"ID":"0a9c0fa0-1913-4627-b146-d4d4f8f2826b","Type":"ContainerDied","Data":"f9fef8b2e5916de7aa705ffb82a2cecbab2e546f0779ff74d430a96bb80baa89"} Nov 03 22:35:21.236385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.235198 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" event={"ID":"2fb0da7d-26b1-4802-b507-b552705b9d30","Type":"ContainerStarted","Data":"04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07"} Nov 03 22:35:21.238046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.237906 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-podinfo" pod "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" (UID: "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:21.279712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.279602 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-token" pod "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" (UID: "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.281462 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.281433 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-kube-api-access-z6tw2" pod "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" (UID: "d0f7c6c6-3025-4adc-9695-0fbfb2ba017e"). InnerVolumeSpecName "kube-api-access-z6tw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.308645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.308601 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.308900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.308882 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.309014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.309000 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6tw2\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-kube-api-access-z6tw2\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.309127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.309114 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.309221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.309209 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.309320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.309309 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.309406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.309392 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.309554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.309524 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.309684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.309672 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.325093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.325070 4903 scope.go:122] "RemoveContainer" containerID="a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb" Nov 03 22:35:21.325749 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:21.325718 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb\": container with ID starting with a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb not found: ID does not exist" containerID="a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb" Nov 03 22:35:21.325860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.325760 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb"} err="failed to get container status \"a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb\": rpc error: code = NotFound desc = could not find container \"a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb\": container with ID starting with a14606fc054cfe45b7e708818d88edd8ebf52a3eae8fd2b33395ae500138e2bb not found: ID does not exist" Nov 03 22:35:21.325860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.325785 4903 scope.go:122] "RemoveContainer" containerID="ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334" Nov 03 22:35:21.329515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.329492 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:35:21.337618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.337583 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:35:21.372525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.354750 4903 scope.go:122] "RemoveContainer" containerID="ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334" Nov 03 22:35:21.372525 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:21.355309 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334\": container with ID starting with ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334 not found: ID does not exist" containerID="ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334" Nov 03 22:35:21.372525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.355355 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334"} err="failed to get container status \"ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334\": rpc error: code = NotFound desc = could not find container \"ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334\": container with ID starting with ae2a9cac867d17818db0156d236c35bf47be0dcd1f4ed89f0243128d42f97334 not found: ID does not exist" Nov 03 22:35:21.380459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.380428 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2"] Nov 03 22:35:21.388945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.388914 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-btjw2"] Nov 03 22:35:21.512265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512232 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-certs\") pod \"621fc439-b1b7-4770-bd4e-529b1fcffe22\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " Nov 03 22:35:21.512471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512276 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-socket\") pod \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " Nov 03 22:35:21.512471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512326 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-token\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-token\") pod \"621fc439-b1b7-4770-bd4e-529b1fcffe22\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " Nov 03 22:35:21.512471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512375 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-envoy\") pod \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " Nov 03 22:35:21.512471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512461 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-credential-socket\") pod \"621fc439-b1b7-4770-bd4e-529b1fcffe22\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " Nov 03 22:35:21.512695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512498 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-kube-api-access-2t4c4\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-kube-api-access-2t4c4\") pod \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " Nov 03 22:35:21.512695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512531 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-certs" pod "621fc439-b1b7-4770-bd4e-529b1fcffe22" (UID: "621fc439-b1b7-4770-bd4e-529b1fcffe22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.512695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512581 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istiod-ca-cert\") pod \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " Nov 03 22:35:21.512695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512621 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-token\") pod \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " Nov 03 22:35:21.512695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512672 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-podinfo\") pod \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " Nov 03 22:35:21.512948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512719 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-kube-api-access-kfxcq\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-kube-api-access-kfxcq\") pod \"621fc439-b1b7-4770-bd4e-529b1fcffe22\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " Nov 03 22:35:21.512948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512824 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-credential-socket" pod "621fc439-b1b7-4770-bd4e-529b1fcffe22" (UID: "621fc439-b1b7-4770-bd4e-529b1fcffe22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.512948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.512901 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-certs\") pod \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " Nov 03 22:35:21.513104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513074 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-socket\") pod \"621fc439-b1b7-4770-bd4e-529b1fcffe22\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " Nov 03 22:35:21.513167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513108 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-data\") pod \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " Nov 03 22:35:21.513167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513146 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621fc439-b1b7-4770-bd4e-529b1fcffe22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621fc439-b1b7-4770-bd4e-529b1fcffe22-istiod-ca-cert\") pod \"621fc439-b1b7-4770-bd4e-529b1fcffe22\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " Nov 03 22:35:21.513264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513214 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-credential-socket\") pod \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\" (UID: \"0a9c0fa0-1913-4627-b146-d4d4f8f2826b\") " Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513380 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-envoy\") pod \"621fc439-b1b7-4770-bd4e-529b1fcffe22\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513424 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-podinfo\") pod \"621fc439-b1b7-4770-bd4e-529b1fcffe22\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513469 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-data\") pod \"621fc439-b1b7-4770-bd4e-529b1fcffe22\" (UID: \"621fc439-b1b7-4770-bd4e-529b1fcffe22\") " Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.514612 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.514636 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513467 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istiod-ca-cert" pod "0a9c0fa0-1913-4627-b146-d4d4f8f2826b" (UID: "0a9c0fa0-1913-4627-b146-d4d4f8f2826b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.514644 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-data" pod "0a9c0fa0-1913-4627-b146-d4d4f8f2826b" (UID: "0a9c0fa0-1913-4627-b146-d4d4f8f2826b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513631 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-socket" pod "0a9c0fa0-1913-4627-b146-d4d4f8f2826b" (UID: "0a9c0fa0-1913-4627-b146-d4d4f8f2826b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.513979 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621fc439-b1b7-4770-bd4e-529b1fcffe22-istiod-ca-cert" pod "621fc439-b1b7-4770-bd4e-529b1fcffe22" (UID: "621fc439-b1b7-4770-bd4e-529b1fcffe22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.514267 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-certs" pod "0a9c0fa0-1913-4627-b146-d4d4f8f2826b" (UID: "0a9c0fa0-1913-4627-b146-d4d4f8f2826b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.514481 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-socket" pod "621fc439-b1b7-4770-bd4e-529b1fcffe22" (UID: "621fc439-b1b7-4770-bd4e-529b1fcffe22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.514865 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-credential-socket" pod "0a9c0fa0-1913-4627-b146-d4d4f8f2826b" (UID: "0a9c0fa0-1913-4627-b146-d4d4f8f2826b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.515374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.515045 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-data" pod "621fc439-b1b7-4770-bd4e-529b1fcffe22" (UID: "621fc439-b1b7-4770-bd4e-529b1fcffe22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.516080 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.515683 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-kube-api-access-kfxcq" pod "621fc439-b1b7-4770-bd4e-529b1fcffe22" (UID: "621fc439-b1b7-4770-bd4e-529b1fcffe22"). InnerVolumeSpecName "kube-api-access-kfxcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.516080 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.515842 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-envoy" pod "0a9c0fa0-1913-4627-b146-d4d4f8f2826b" (UID: "0a9c0fa0-1913-4627-b146-d4d4f8f2826b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.516180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.516121 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-kube-api-access-2t4c4" pod "0a9c0fa0-1913-4627-b146-d4d4f8f2826b" (UID: "0a9c0fa0-1913-4627-b146-d4d4f8f2826b"). InnerVolumeSpecName "kube-api-access-2t4c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.516276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.516254 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-podinfo" pod "0a9c0fa0-1913-4627-b146-d4d4f8f2826b" (UID: "0a9c0fa0-1913-4627-b146-d4d4f8f2826b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:21.516276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.516261 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-token" pod "0a9c0fa0-1913-4627-b146-d4d4f8f2826b" (UID: "0a9c0fa0-1913-4627-b146-d4d4f8f2826b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.517557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.517527 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-podinfo" pod "621fc439-b1b7-4770-bd4e-529b1fcffe22" (UID: "621fc439-b1b7-4770-bd4e-529b1fcffe22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:21.517688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.517656 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-token" pod "621fc439-b1b7-4770-bd4e-529b1fcffe22" (UID: "621fc439-b1b7-4770-bd4e-529b1fcffe22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:21.517790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.517759 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-envoy" pod "621fc439-b1b7-4770-bd4e-529b1fcffe22" (UID: "621fc439-b1b7-4770-bd4e-529b1fcffe22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:21.609689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.609654 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84"] Nov 03 22:35:21.614720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.614692 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-qzh84"] Nov 03 22:35:21.615822 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615794 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.615822 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615822 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615836 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615848 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t4c4\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-kube-api-access-2t4c4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615862 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615876 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615889 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615929 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfxcq\" (UniqueName: \"kubernetes.io/projected/621fc439-b1b7-4770-bd4e-529b1fcffe22-kube-api-access-kfxcq\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615941 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615956 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615969 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615983 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621fc439-b1b7-4770-bd4e-529b1fcffe22-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.615996 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9c0fa0-1913-4627-b146-d4d4f8f2826b-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.616008 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.616021 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.616400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.616034 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621fc439-b1b7-4770-bd4e-529b1fcffe22-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:21.763319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.741150 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" path="/var/lib/kubelet/pods/8ca6fd32-0245-4ea1-ae2c-81512aa4e979/volumes" Nov 03 22:35:21.763319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.741978 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" path="/var/lib/kubelet/pods/d0f7c6c6-3025-4adc-9695-0fbfb2ba017e/volumes" Nov 03 22:35:21.865780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.865665 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:35:21.865950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.865806 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:35:21.865950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:21.865862 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 22:35:22.270640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.270608 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" Nov 03 22:35:22.271111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.270636 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26" event={"ID":"0a9c0fa0-1913-4627-b146-d4d4f8f2826b","Type":"ContainerDied","Data":"42148a39655917176fa94b8994c46a73e128b14b4b0718ca743522f20b1a4be2"} Nov 03 22:35:22.271111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.270689 4903 scope.go:122] "RemoveContainer" containerID="f9fef8b2e5916de7aa705ffb82a2cecbab2e546f0779ff74d430a96bb80baa89" Nov 03 22:35:22.275275 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.275252 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" Nov 03 22:35:22.275393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.275286 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh" event={"ID":"621fc439-b1b7-4770-bd4e-529b1fcffe22","Type":"ContainerDied","Data":"c2d91bba862f537b5f69f8152d35481438bb310afca08ca101042b9c89632174"} Nov 03 22:35:22.290849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.290827 4903 scope.go:122] "RemoveContainer" containerID="02331e20d94cda190bdf624b4bd40bf0be04909881b04b3f1947908389166c11" Nov 03 22:35:22.320903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.320875 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26"] Nov 03 22:35:22.325838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.325809 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-hdk26"] Nov 03 22:35:22.370662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.370609 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh"] Nov 03 22:35:22.377773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.377744 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-nqgqh"] Nov 03 22:35:22.865316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.863726 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:35:22.865316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:22.863792 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 22:35:23.805590 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:23.803718 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" path="/var/lib/kubelet/pods/0a9c0fa0-1913-4627-b146-d4d4f8f2826b/volumes" Nov 03 22:35:23.805590 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:23.804408 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" path="/var/lib/kubelet/pods/621fc439-b1b7-4770-bd4e-529b1fcffe22/volumes" Nov 03 22:35:23.866721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:23.866672 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:35:23.867007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:23.866980 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 22:35:24.863316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.862700 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:35:24.863316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.862763 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 22:35:24.883542 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.883508 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-5842ac"] Nov 03 22:35:24.884338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884323 4903 cpu_manager.go:414] "containerMap: removing container" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" Nov 03 22:35:24.884394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884340 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" Nov 03 22:35:24.884394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884361 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" Nov 03 22:35:24.884394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884367 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" Nov 03 22:35:24.884394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884391 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" Nov 03 22:35:24.884516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884397 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" Nov 03 22:35:24.884516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884404 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" Nov 03 22:35:24.884516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884409 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" Nov 03 22:35:24.884602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884542 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f7c6c6-3025-4adc-9695-0fbfb2ba017e" containerName="istio-proxy" Nov 03 22:35:24.884602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884553 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca6fd32-0245-4ea1-ae2c-81512aa4e979" containerName="istio-proxy" Nov 03 22:35:24.884602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884566 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="621fc439-b1b7-4770-bd4e-529b1fcffe22" containerName="istio-proxy" Nov 03 22:35:24.884602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.884576 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9c0fa0-1913-4627-b146-d4d4f8f2826b" containerName="istio-proxy" Nov 03 22:35:24.889130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.889111 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-5842ac" Nov 03 22:35:24.903190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.903163 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-5842ac"] Nov 03 22:35:24.907481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:24.907460 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-khmwt\"" Nov 03 22:35:25.014935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:25.014906 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rsh\" (UniqueName: \"kubernetes.io/projected/758c1e41-8fee-413f-8e45-b367489296f3-kube-api-access-j8rsh\") pod \"test-curl-7257-5842ac\" (UID: \"758c1e41-8fee-413f-8e45-b367489296f3\") " pod="opendatahub/test-curl-7257-5842ac" Nov 03 22:35:25.117077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:25.116986 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rsh\" (UniqueName: \"kubernetes.io/projected/758c1e41-8fee-413f-8e45-b367489296f3-kube-api-access-j8rsh\") pod \"test-curl-7257-5842ac\" (UID: \"758c1e41-8fee-413f-8e45-b367489296f3\") " pod="opendatahub/test-curl-7257-5842ac" Nov 03 22:35:25.139000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:25.138958 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rsh\" (UniqueName: \"kubernetes.io/projected/758c1e41-8fee-413f-8e45-b367489296f3-kube-api-access-j8rsh\") pod \"test-curl-7257-5842ac\" (UID: \"758c1e41-8fee-413f-8e45-b367489296f3\") " pod="opendatahub/test-curl-7257-5842ac" Nov 03 22:35:25.203404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:25.203373 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-5842ac" Nov 03 22:35:25.575194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:25.575170 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-5842ac"] Nov 03 22:35:25.580554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:25.580532 4903 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:35:25.865832 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:25.865733 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Nov 03 22:35:25.865832 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:25.865809 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Nov 03 22:35:26.331962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:26.331921 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-5842ac" event={"ID":"758c1e41-8fee-413f-8e45-b367489296f3","Type":"ContainerStarted","Data":"880706e523337a55873c99c6fe8ebfd1e875cd5b265922a779c75196886ee35b"} Nov 03 22:35:26.868263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:26.868227 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:35:26.868935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:26.868914 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:35:26.869812 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:26.869794 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 22:35:28.368753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:28.368716 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-5842ac" event={"ID":"758c1e41-8fee-413f-8e45-b367489296f3","Type":"ContainerStarted","Data":"f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7"} Nov 03 22:35:28.406135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:28.406065 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-5842ac" podStartSLOduration=2.162020941 podStartE2EDuration="4.406045573s" totalImagesPullingTime="2.244024632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:28.404834889 +0000 UTC m=+1354.808775867" watchObservedRunningTime="2026-11-03 22:35:28.406045573 +0000 UTC m=+1354.809986548" Nov 03 22:35:29.289605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.289558 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-5842ac"] Nov 03 22:35:29.294319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.294274 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-5842ac"] Nov 03 22:35:29.963150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.963110 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc"] Nov 03 22:35:29.964446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.964423 4903 cpu_manager.go:414] "containerMap: removing container" podUID="758c1e41-8fee-413f-8e45-b367489296f3" containerName="test-curl-7257-5842ac" Nov 03 22:35:29.964500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.964448 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758c1e41-8fee-413f-8e45-b367489296f3" containerName="test-curl-7257-5842ac" Nov 03 22:35:29.964756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.964744 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="758c1e41-8fee-413f-8e45-b367489296f3" containerName="test-curl-7257-5842ac" Nov 03 22:35:29.968282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.968261 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:29.984359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.984317 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc"] Nov 03 22:35:29.986835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.986805 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:35:29.986835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.986824 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:35:29.987044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.986815 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:35:29.987044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:29.986902 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:35:30.044650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.044610 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.044852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.044664 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27h7\" (UniqueName: \"kubernetes.io/projected/d96c50f9-d2c2-4acc-8900-562926ab8f18-kube-api-access-k27h7\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.045060 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.045030 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.146760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.146719 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27h7\" (UniqueName: \"kubernetes.io/projected/d96c50f9-d2c2-4acc-8900-562926ab8f18-kube-api-access-k27h7\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.147040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.147022 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.147219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.147202 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.147732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.147705 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.149699 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.149679 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.168189 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.168166 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27h7\" (UniqueName: \"kubernetes.io/projected/d96c50f9-d2c2-4acc-8900-562926ab8f18-kube-api-access-k27h7\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.198534 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.198506 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg"] Nov 03 22:35:30.198758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.198733 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" containerID="cri-o://2054cff75d0acc5785f8e504972c6305b01738c81d67615a4391c12299e0b304" gracePeriod=30 Nov 03 22:35:30.292002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.291924 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:30.394380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.394216 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-5842ac" podUID="758c1e41-8fee-413f-8e45-b367489296f3" containerName="test-curl-7257-5842ac" containerID="cri-o://f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7" gracePeriod=2 Nov 03 22:35:30.612464 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:35:30.612432 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96c50f9_d2c2_4acc_8900_562926ab8f18.slice/crio-722ed901e1004d03579a4a90a739d439de4cbebb93442048b6477e656756145d WatchSource:0}: Error finding container 722ed901e1004d03579a4a90a739d439de4cbebb93442048b6477e656756145d: Status 404 returned error can't find the container with id 722ed901e1004d03579a4a90a739d439de4cbebb93442048b6477e656756145d Nov 03 22:35:30.613229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.613194 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc"] Nov 03 22:35:30.695131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.695098 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q"] Nov 03 22:35:30.695381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.695352 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" containerID="cri-o://db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb" gracePeriod=30 Nov 03 22:35:30.872960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.872874 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld"] Nov 03 22:35:30.873130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.873104 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" containerID="cri-o://f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4" gracePeriod=30 Nov 03 22:35:30.878339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.878304 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq"] Nov 03 22:35:30.878507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:30.878486 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" containerID="cri-o://f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1" gracePeriod=30 Nov 03 22:35:31.030363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:31.030288 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q"] Nov 03 22:35:31.034637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:31.034605 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fb51a2-cdk2q"] Nov 03 22:35:31.409144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:31.409056 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" event={"ID":"d96c50f9-d2c2-4acc-8900-562926ab8f18","Type":"ContainerStarted","Data":"6524ea3bb37447aed04b9dd1bcde9406a583327ceda8f0a7abead7f0b310c811"} Nov 03 22:35:31.409144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:31.409119 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" event={"ID":"d96c50f9-d2c2-4acc-8900-562926ab8f18","Type":"ContainerStarted","Data":"722ed901e1004d03579a4a90a739d439de4cbebb93442048b6477e656756145d"} Nov 03 22:35:31.450014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:31.449966 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" podStartSLOduration=2.449942816 podStartE2EDuration="2.449942816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:31.445336395 +0000 UTC m=+1357.849277369" watchObservedRunningTime="2026-11-03 22:35:31.449942816 +0000 UTC m=+1357.853883790" Nov 03 22:35:31.736398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:31.736365 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818eafd8-baaa-4905-b3cc-237c00fdcb73" path="/var/lib/kubelet/pods/818eafd8-baaa-4905-b3cc-237c00fdcb73/volumes" Nov 03 22:35:32.420969 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:32.420931 4903 generic.go:356] "Generic (PLEG): container finished" podID="d96c50f9-d2c2-4acc-8900-562926ab8f18" containerID="6524ea3bb37447aed04b9dd1bcde9406a583327ceda8f0a7abead7f0b310c811" exitCode=0 Nov 03 22:35:32.421399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:32.421064 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" event={"ID":"d96c50f9-d2c2-4acc-8900-562926ab8f18","Type":"ContainerDied","Data":"6524ea3bb37447aed04b9dd1bcde9406a583327ceda8f0a7abead7f0b310c811"} Nov 03 22:35:32.651510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:32.651485 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-5842ac" Nov 03 22:35:32.794976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:32.794948 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758c1e41-8fee-413f-8e45-b367489296f3-kube-api-access-j8rsh\" (UniqueName: \"kubernetes.io/projected/758c1e41-8fee-413f-8e45-b367489296f3-kube-api-access-j8rsh\") pod \"758c1e41-8fee-413f-8e45-b367489296f3\" (UID: \"758c1e41-8fee-413f-8e45-b367489296f3\") " Nov 03 22:35:32.797111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:32.797077 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758c1e41-8fee-413f-8e45-b367489296f3-kube-api-access-j8rsh" pod "758c1e41-8fee-413f-8e45-b367489296f3" (UID: "758c1e41-8fee-413f-8e45-b367489296f3"). InnerVolumeSpecName "kube-api-access-j8rsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:32.897684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:32.897649 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rsh\" (UniqueName: \"kubernetes.io/projected/758c1e41-8fee-413f-8e45-b367489296f3-kube-api-access-j8rsh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:33.438161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.438119 4903 generic.go:356] "Generic (PLEG): container finished" podID="758c1e41-8fee-413f-8e45-b367489296f3" containerID="f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7" exitCode=137 Nov 03 22:35:33.440450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.439898 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-5842ac" Nov 03 22:35:33.448771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.448237 4903 scope.go:122] "RemoveContainer" containerID="f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7" Nov 03 22:35:33.545259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.542218 4903 scope.go:122] "RemoveContainer" containerID="f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7" Nov 03 22:35:33.551091 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:33.550952 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7\": container with ID starting with f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7 not found: ID does not exist" containerID="f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7" Nov 03 22:35:33.551091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.551001 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7"} err="failed to get container status \"f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7\": rpc error: code = NotFound desc = could not find container \"f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7\": container with ID starting with f86837c50d4463f9f6115a744d6fc1f12691a393a0a23601b3e80dba3db8e8a7 not found: ID does not exist" Nov 03 22:35:33.608874 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:33.608836 4903 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-5842ac\" is forbidden: User \"system:node:ip-10-0-133-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-193.ec2.internal' and this object" podUID="758c1e41-8fee-413f-8e45-b367489296f3" pod="opendatahub/test-curl-7257-5842ac" Nov 03 22:35:33.677637 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:33.677055 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod758c1e41_8fee_413f_8e45_b367489296f3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod758c1e41_8fee_413f_8e45_b367489296f3.slice/crio-880706e523337a55873c99c6fe8ebfd1e875cd5b265922a779c75196886ee35b\": RecentStats: unable to find data in memory cache]" Nov 03 22:35:33.751375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.751209 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758c1e41-8fee-413f-8e45-b367489296f3" path="/var/lib/kubelet/pods/758c1e41-8fee-413f-8e45-b367489296f3/volumes" Nov 03 22:35:33.782170 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:33.782123 4903 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-5842ac\" is forbidden: User \"system:node:ip-10-0-133-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-193.ec2.internal' and this object" podUID="758c1e41-8fee-413f-8e45-b367489296f3" pod="opendatahub/test-curl-7257-5842ac" Nov 03 22:35:33.847612 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.847580 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:33.960723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.960680 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-openshift-ca\") pod \"d96c50f9-d2c2-4acc-8900-562926ab8f18\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " Nov 03 22:35:33.960923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.960779 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96c50f9-d2c2-4acc-8900-562926ab8f18-kube-api-access-k27h7\" (UniqueName: \"kubernetes.io/projected/d96c50f9-d2c2-4acc-8900-562926ab8f18-kube-api-access-k27h7\") pod \"d96c50f9-d2c2-4acc-8900-562926ab8f18\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " Nov 03 22:35:33.960999 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.960977 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-xks-ca\") pod \"d96c50f9-d2c2-4acc-8900-562926ab8f18\" (UID: \"d96c50f9-d2c2-4acc-8900-562926ab8f18\") " Nov 03 22:35:33.972285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.972229 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96c50f9-d2c2-4acc-8900-562926ab8f18-kube-api-access-k27h7" pod "d96c50f9-d2c2-4acc-8900-562926ab8f18" (UID: "d96c50f9-d2c2-4acc-8900-562926ab8f18"). InnerVolumeSpecName "kube-api-access-k27h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:33.975370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.974535 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-openshift-ca" pod "d96c50f9-d2c2-4acc-8900-562926ab8f18" (UID: "d96c50f9-d2c2-4acc-8900-562926ab8f18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:33.994647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:33.994563 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-xks-ca" pod "d96c50f9-d2c2-4acc-8900-562926ab8f18" (UID: "d96c50f9-d2c2-4acc-8900-562926ab8f18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:35:34.002557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:34.002420 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:34.002557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:34.002457 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c50f9-d2c2-4acc-8900-562926ab8f18-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:34.002557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:34.002478 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27h7\" (UniqueName: \"kubernetes.io/projected/d96c50f9-d2c2-4acc-8900-562926ab8f18-kube-api-access-k27h7\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:34.516962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:34.516924 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" event={"ID":"d96c50f9-d2c2-4acc-8900-562926ab8f18","Type":"ContainerDied","Data":"722ed901e1004d03579a4a90a739d439de4cbebb93442048b6477e656756145d"} Nov 03 22:35:34.518340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:34.516894 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc" Nov 03 22:35:34.518510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:34.518427 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="722ed901e1004d03579a4a90a739d439de4cbebb93442048b6477e656756145d" Nov 03 22:35:35.531644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.531569 4903 generic.go:356] "Generic (PLEG): container finished" podID="00b28bb3-915a-4303-a100-84b16be20ebc" containerID="2054cff75d0acc5785f8e504972c6305b01738c81d67615a4391c12299e0b304" exitCode=0 Nov 03 22:35:35.532064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.531644 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" event={"ID":"00b28bb3-915a-4303-a100-84b16be20ebc","Type":"ContainerDied","Data":"2054cff75d0acc5785f8e504972c6305b01738c81d67615a4391c12299e0b304"} Nov 03 22:35:35.532064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.531694 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" event={"ID":"00b28bb3-915a-4303-a100-84b16be20ebc","Type":"ContainerDied","Data":"23523c692ceba2cc69fd2483032ef95eec732e6c288df6eef69283f9cbf05a07"} Nov 03 22:35:35.532064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.531711 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23523c692ceba2cc69fd2483032ef95eec732e6c288df6eef69283f9cbf05a07" Nov 03 22:35:35.532901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.532883 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:35:35.665444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.665407 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-istio-token\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-istio-token\") pod \"00b28bb3-915a-4303-a100-84b16be20ebc\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " Nov 03 22:35:35.665645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.665521 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-certs\") pod \"00b28bb3-915a-4303-a100-84b16be20ebc\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " Nov 03 22:35:35.666041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.665843 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-kube-api-access-pwpsp\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-kube-api-access-pwpsp\") pod \"00b28bb3-915a-4303-a100-84b16be20ebc\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " Nov 03 22:35:35.666041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.665941 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-data\") pod \"00b28bb3-915a-4303-a100-84b16be20ebc\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " Nov 03 22:35:35.666041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.666008 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-socket\") pod \"00b28bb3-915a-4303-a100-84b16be20ebc\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " Nov 03 22:35:35.666274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.666054 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b28bb3-915a-4303-a100-84b16be20ebc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b28bb3-915a-4303-a100-84b16be20ebc-istiod-ca-cert\") pod \"00b28bb3-915a-4303-a100-84b16be20ebc\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " Nov 03 22:35:35.666274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.666110 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-credential-socket\") pod \"00b28bb3-915a-4303-a100-84b16be20ebc\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " Nov 03 22:35:35.666274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.666142 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00b28bb3-915a-4303-a100-84b16be20ebc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b28bb3-915a-4303-a100-84b16be20ebc-istio-podinfo\") pod \"00b28bb3-915a-4303-a100-84b16be20ebc\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " Nov 03 22:35:35.666274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.666174 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-envoy\") pod \"00b28bb3-915a-4303-a100-84b16be20ebc\" (UID: \"00b28bb3-915a-4303-a100-84b16be20ebc\") " Nov 03 22:35:35.666574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.666554 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-socket" pod "00b28bb3-915a-4303-a100-84b16be20ebc" (UID: "00b28bb3-915a-4303-a100-84b16be20ebc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:35.667225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.666685 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-certs" pod "00b28bb3-915a-4303-a100-84b16be20ebc" (UID: "00b28bb3-915a-4303-a100-84b16be20ebc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:35.667225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.667093 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:35.667697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.667478 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-data" pod "00b28bb3-915a-4303-a100-84b16be20ebc" (UID: "00b28bb3-915a-4303-a100-84b16be20ebc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:35.667697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.667504 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b28bb3-915a-4303-a100-84b16be20ebc-istiod-ca-cert" pod "00b28bb3-915a-4303-a100-84b16be20ebc" (UID: "00b28bb3-915a-4303-a100-84b16be20ebc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:35.667846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.667790 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-credential-socket" pod "00b28bb3-915a-4303-a100-84b16be20ebc" (UID: "00b28bb3-915a-4303-a100-84b16be20ebc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:35.670215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.670190 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-istio-token" pod "00b28bb3-915a-4303-a100-84b16be20ebc" (UID: "00b28bb3-915a-4303-a100-84b16be20ebc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:35.670364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.670231 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-kube-api-access-pwpsp" pod "00b28bb3-915a-4303-a100-84b16be20ebc" (UID: "00b28bb3-915a-4303-a100-84b16be20ebc"). InnerVolumeSpecName "kube-api-access-pwpsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:35.674730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.674701 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00b28bb3-915a-4303-a100-84b16be20ebc-istio-podinfo" pod "00b28bb3-915a-4303-a100-84b16be20ebc" (UID: "00b28bb3-915a-4303-a100-84b16be20ebc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:35.674833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.674766 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-envoy" pod "00b28bb3-915a-4303-a100-84b16be20ebc" (UID: "00b28bb3-915a-4303-a100-84b16be20ebc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:35.769336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.769285 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpsp\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-kube-api-access-pwpsp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:35.769336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.769334 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:35.769516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.769349 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00b28bb3-915a-4303-a100-84b16be20ebc-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:35.769516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.769363 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:35.769516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.769379 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00b28bb3-915a-4303-a100-84b16be20ebc-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:35.769516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.769403 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:35.769516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.769415 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00b28bb3-915a-4303-a100-84b16be20ebc-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:35.769516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:35.769428 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00b28bb3-915a-4303-a100-84b16be20ebc-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.019805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.019747 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:35:36.189721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.189685 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-data\") pod \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " Nov 03 22:35:36.190167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.190144 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-podinfo\") pod \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " Nov 03 22:35:36.190288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.190273 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istiod-ca-cert\") pod \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " Nov 03 22:35:36.190502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.190488 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-kube-api-access-6q77t\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-kube-api-access-6q77t\") pod \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " Nov 03 22:35:36.190614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.190599 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-token\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-token\") pod \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " Nov 03 22:35:36.190756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.190744 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-certs\") pod \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " Nov 03 22:35:36.190991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.190976 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-envoy\") pod \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " Nov 03 22:35:36.191210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.191189 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-credential-socket\") pod \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " Nov 03 22:35:36.191423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.191402 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-socket\") pod \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\" (UID: \"0e6096f0-23ba-4115-9f27-84fbd9765f1b\") " Nov 03 22:35:36.192632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.192609 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-socket" pod "0e6096f0-23ba-4115-9f27-84fbd9765f1b" (UID: "0e6096f0-23ba-4115-9f27-84fbd9765f1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.192958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.192941 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-data" pod "0e6096f0-23ba-4115-9f27-84fbd9765f1b" (UID: "0e6096f0-23ba-4115-9f27-84fbd9765f1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.194641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.193961 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-certs" pod "0e6096f0-23ba-4115-9f27-84fbd9765f1b" (UID: "0e6096f0-23ba-4115-9f27-84fbd9765f1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.194641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.194345 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istiod-ca-cert" pod "0e6096f0-23ba-4115-9f27-84fbd9765f1b" (UID: "0e6096f0-23ba-4115-9f27-84fbd9765f1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:36.194641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.194603 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-credential-socket" pod "0e6096f0-23ba-4115-9f27-84fbd9765f1b" (UID: "0e6096f0-23ba-4115-9f27-84fbd9765f1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.197685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.197646 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-envoy" pod "0e6096f0-23ba-4115-9f27-84fbd9765f1b" (UID: "0e6096f0-23ba-4115-9f27-84fbd9765f1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.208825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.208791 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-token" pod "0e6096f0-23ba-4115-9f27-84fbd9765f1b" (UID: "0e6096f0-23ba-4115-9f27-84fbd9765f1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.209009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.208959 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-podinfo" pod "0e6096f0-23ba-4115-9f27-84fbd9765f1b" (UID: "0e6096f0-23ba-4115-9f27-84fbd9765f1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:36.216356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.214685 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-kube-api-access-6q77t" pod "0e6096f0-23ba-4115-9f27-84fbd9765f1b" (UID: "0e6096f0-23ba-4115-9f27-84fbd9765f1b"). InnerVolumeSpecName "kube-api-access-6q77t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.300992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.300963 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.300992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.300991 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.301190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.301003 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.301190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.301018 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q77t\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-kube-api-access-6q77t\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.301190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.301029 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.301190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.301041 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.301190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.301052 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.301190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.301064 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.301190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.301075 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6096f0-23ba-4115-9f27-84fbd9765f1b-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.350381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.350344 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:35:36.353587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.353065 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:35:36.503712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.503676 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-certs\") pod \"0ace52e6-3c28-4087-b7c5-81685c6533db\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " Nov 03 22:35:36.503894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.503774 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-data\") pod \"0ace52e6-3c28-4087-b7c5-81685c6533db\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " Nov 03 22:35:36.503894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.503806 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-kube-api-access-xzgbx\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-kube-api-access-xzgbx\") pod \"04e059a1-1e9b-4a02-a988-b0744036103e\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " Nov 03 22:35:36.503894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.503892 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-podinfo\") pod \"0ace52e6-3c28-4087-b7c5-81685c6533db\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " Nov 03 22:35:36.504055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.503931 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-token\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-token\") pod \"0ace52e6-3c28-4087-b7c5-81685c6533db\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " Nov 03 22:35:36.504055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.503966 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-certs" pod "0ace52e6-3c28-4087-b7c5-81685c6533db" (UID: "0ace52e6-3c28-4087-b7c5-81685c6533db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.504055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.503985 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-envoy\") pod \"0ace52e6-3c28-4087-b7c5-81685c6533db\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " Nov 03 22:35:36.504055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504029 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ace52e6-3c28-4087-b7c5-81685c6533db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace52e6-3c28-4087-b7c5-81685c6533db-istiod-ca-cert\") pod \"0ace52e6-3c28-4087-b7c5-81685c6533db\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " Nov 03 22:35:36.504263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504059 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-data" pod "0ace52e6-3c28-4087-b7c5-81685c6533db" (UID: "0ace52e6-3c28-4087-b7c5-81685c6533db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.504263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504074 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-socket\") pod \"04e059a1-1e9b-4a02-a988-b0744036103e\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " Nov 03 22:35:36.504263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504111 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-istio-token\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-istio-token\") pod \"04e059a1-1e9b-4a02-a988-b0744036103e\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " Nov 03 22:35:36.504263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504165 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-certs\") pod \"04e059a1-1e9b-4a02-a988-b0744036103e\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " Nov 03 22:35:36.504263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504201 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-credential-socket\") pod \"0ace52e6-3c28-4087-b7c5-81685c6533db\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " Nov 03 22:35:36.504263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504252 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-envoy\") pod \"04e059a1-1e9b-4a02-a988-b0744036103e\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " Nov 03 22:35:36.504600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504340 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-socket\") pod \"0ace52e6-3c28-4087-b7c5-81685c6533db\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " Nov 03 22:35:36.504600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504377 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e059a1-1e9b-4a02-a988-b0744036103e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e059a1-1e9b-4a02-a988-b0744036103e-istiod-ca-cert\") pod \"04e059a1-1e9b-4a02-a988-b0744036103e\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " Nov 03 22:35:36.504600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504379 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-socket" pod "04e059a1-1e9b-4a02-a988-b0744036103e" (UID: "04e059a1-1e9b-4a02-a988-b0744036103e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.504600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504405 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ace52e6-3c28-4087-b7c5-81685c6533db-istiod-ca-cert" pod "0ace52e6-3c28-4087-b7c5-81685c6533db" (UID: "0ace52e6-3c28-4087-b7c5-81685c6533db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:36.504600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504506 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-certs" pod "04e059a1-1e9b-4a02-a988-b0744036103e" (UID: "04e059a1-1e9b-4a02-a988-b0744036103e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.504600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504515 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-credential-socket\") pod \"04e059a1-1e9b-4a02-a988-b0744036103e\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " Nov 03 22:35:36.504600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504572 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-kube-api-access-496tf\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-kube-api-access-496tf\") pod \"0ace52e6-3c28-4087-b7c5-81685c6533db\" (UID: \"0ace52e6-3c28-4087-b7c5-81685c6533db\") " Nov 03 22:35:36.504937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504649 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-data\") pod \"04e059a1-1e9b-4a02-a988-b0744036103e\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " Nov 03 22:35:36.504937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504716 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04e059a1-1e9b-4a02-a988-b0744036103e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e059a1-1e9b-4a02-a988-b0744036103e-istio-podinfo\") pod \"04e059a1-1e9b-4a02-a988-b0744036103e\" (UID: \"04e059a1-1e9b-4a02-a988-b0744036103e\") " Nov 03 22:35:36.504937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504731 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-credential-socket" pod "04e059a1-1e9b-4a02-a988-b0744036103e" (UID: "04e059a1-1e9b-4a02-a988-b0744036103e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.505076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504938 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-socket" pod "0ace52e6-3c28-4087-b7c5-81685c6533db" (UID: "0ace52e6-3c28-4087-b7c5-81685c6533db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.505076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.504952 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-credential-socket" pod "0ace52e6-3c28-4087-b7c5-81685c6533db" (UID: "0ace52e6-3c28-4087-b7c5-81685c6533db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.505746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.505111 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-data" pod "04e059a1-1e9b-4a02-a988-b0744036103e" (UID: "04e059a1-1e9b-4a02-a988-b0744036103e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.505746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.505435 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e059a1-1e9b-4a02-a988-b0744036103e-istiod-ca-cert" pod "04e059a1-1e9b-4a02-a988-b0744036103e" (UID: "04e059a1-1e9b-4a02-a988-b0744036103e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:36.506896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.506868 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-podinfo" pod "0ace52e6-3c28-4087-b7c5-81685c6533db" (UID: "0ace52e6-3c28-4087-b7c5-81685c6533db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:36.507173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.507150 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-token" pod "0ace52e6-3c28-4087-b7c5-81685c6533db" (UID: "0ace52e6-3c28-4087-b7c5-81685c6533db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.507407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.507383 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-istio-token" pod "04e059a1-1e9b-4a02-a988-b0744036103e" (UID: "04e059a1-1e9b-4a02-a988-b0744036103e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.507497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.507454 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04e059a1-1e9b-4a02-a988-b0744036103e-istio-podinfo" pod "04e059a1-1e9b-4a02-a988-b0744036103e" (UID: "04e059a1-1e9b-4a02-a988-b0744036103e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:35:36.507702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.507683 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-envoy" pod "0ace52e6-3c28-4087-b7c5-81685c6533db" (UID: "0ace52e6-3c28-4087-b7c5-81685c6533db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.507769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.507748 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-kube-api-access-xzgbx" pod "04e059a1-1e9b-4a02-a988-b0744036103e" (UID: "04e059a1-1e9b-4a02-a988-b0744036103e"). InnerVolumeSpecName "kube-api-access-xzgbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.508119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508102 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508122 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508133 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508142 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508155 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ace52e6-3c28-4087-b7c5-81685c6533db-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508165 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508173 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508182 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508190 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508198 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e059a1-1e9b-4a02-a988-b0744036103e-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.508394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508324 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-kube-api-access-496tf" pod "0ace52e6-3c28-4087-b7c5-81685c6533db" (UID: "0ace52e6-3c28-4087-b7c5-81685c6533db"). InnerVolumeSpecName "kube-api-access-496tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:36.508776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.508757 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-envoy" pod "04e059a1-1e9b-4a02-a988-b0744036103e" (UID: "04e059a1-1e9b-4a02-a988-b0744036103e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:35:36.543396 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.543366 4903 generic.go:356] "Generic (PLEG): container finished" podID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerID="f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4" exitCode=0 Nov 03 22:35:36.543849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.543420 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" Nov 03 22:35:36.543849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.543455 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" event={"ID":"0ace52e6-3c28-4087-b7c5-81685c6533db","Type":"ContainerDied","Data":"f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4"} Nov 03 22:35:36.543849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.543514 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld" event={"ID":"0ace52e6-3c28-4087-b7c5-81685c6533db","Type":"ContainerDied","Data":"0c33646e10b7eee12015fbdba94d9301a18726bc20efab12415621f4fd49dc77"} Nov 03 22:35:36.543849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.543546 4903 scope.go:122] "RemoveContainer" containerID="f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4" Nov 03 22:35:36.545220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.545194 4903 generic.go:356] "Generic (PLEG): container finished" podID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerID="db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb" exitCode=0 Nov 03 22:35:36.545383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.545247 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" Nov 03 22:35:36.545383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.545279 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" event={"ID":"0e6096f0-23ba-4115-9f27-84fbd9765f1b","Type":"ContainerDied","Data":"db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb"} Nov 03 22:35:36.545383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.545355 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q" event={"ID":"0e6096f0-23ba-4115-9f27-84fbd9765f1b","Type":"ContainerDied","Data":"ae2d2ac0a6aca9d98280d2d2491515e1c5bd6e4340f274499286e2a63359f843"} Nov 03 22:35:36.548576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.548555 4903 generic.go:356] "Generic (PLEG): container finished" podID="04e059a1-1e9b-4a02-a988-b0744036103e" containerID="f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1" exitCode=0 Nov 03 22:35:36.548685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.548595 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" Nov 03 22:35:36.548685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.548630 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" event={"ID":"04e059a1-1e9b-4a02-a988-b0744036103e","Type":"ContainerDied","Data":"f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1"} Nov 03 22:35:36.548685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.548674 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq" event={"ID":"04e059a1-1e9b-4a02-a988-b0744036103e","Type":"ContainerDied","Data":"0d8166336d8125b4a76920c5f35e7aea537ba95965338c316584a7dceab5c550"} Nov 03 22:35:36.548922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.548767 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg" Nov 03 22:35:36.562504 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.562238 4903 scope.go:122] "RemoveContainer" containerID="f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4" Nov 03 22:35:36.562852 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:36.562823 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4\": container with ID starting with f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4 not found: ID does not exist" containerID="f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4" Nov 03 22:35:36.563155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.562861 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4"} err="failed to get container status \"f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4\": rpc error: code = NotFound desc = could not find container \"f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4\": container with ID starting with f005162cf5d204bedf8e762516d96a473dc2975106e2a5e4392ae7892628eeb4 not found: ID does not exist" Nov 03 22:35:36.563155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.562886 4903 scope.go:122] "RemoveContainer" containerID="db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb" Nov 03 22:35:36.630631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.630481 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzgbx\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-kube-api-access-xzgbx\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.630631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.630515 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.630631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.630530 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.630631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.630545 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ace52e6-3c28-4087-b7c5-81685c6533db-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.630631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.630558 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e059a1-1e9b-4a02-a988-b0744036103e-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.630631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.630570 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e059a1-1e9b-4a02-a988-b0744036103e-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.630631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.630583 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-496tf\" (UniqueName: \"kubernetes.io/projected/0ace52e6-3c28-4087-b7c5-81685c6533db-kube-api-access-496tf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.630631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.630596 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e059a1-1e9b-4a02-a988-b0744036103e-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:36.699389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.696500 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg"] Nov 03 22:35:36.708672 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.708639 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-rdndg"] Nov 03 22:35:36.718811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.717849 4903 scope.go:122] "RemoveContainer" containerID="db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb" Nov 03 22:35:36.720470 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:36.720096 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb\": container with ID starting with db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb not found: ID does not exist" containerID="db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb" Nov 03 22:35:36.720470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.720134 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb"} err="failed to get container status \"db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb\": rpc error: code = NotFound desc = could not find container \"db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb\": container with ID starting with db58cf133671ef49bdce5e5db8242ef94ecf38846f41566bba21b873abf8eabb not found: ID does not exist" Nov 03 22:35:36.720470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.720161 4903 scope.go:122] "RemoveContainer" containerID="f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1" Nov 03 22:35:36.739652 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.739366 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q"] Nov 03 22:35:36.766606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.765856 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-vgf7q"] Nov 03 22:35:36.782844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.782174 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld"] Nov 03 22:35:36.782844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.782230 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-gwqld"] Nov 03 22:35:36.782844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.782759 4903 scope.go:122] "RemoveContainer" containerID="f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1" Nov 03 22:35:36.783536 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:36.783510 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1\": container with ID starting with f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1 not found: ID does not exist" containerID="f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1" Nov 03 22:35:36.783670 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.783545 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1"} err="failed to get container status \"f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1\": rpc error: code = NotFound desc = could not find container \"f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1\": container with ID starting with f07e0b20bcdb7cea46a31a976f8a38015f67834019225894723cc596963395e1 not found: ID does not exist" Nov 03 22:35:36.805532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.805488 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq"] Nov 03 22:35:36.810926 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:36.810896 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-hz2jq"] Nov 03 22:35:37.740964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:37.740926 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" path="/var/lib/kubelet/pods/00b28bb3-915a-4303-a100-84b16be20ebc/volumes" Nov 03 22:35:37.741731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:37.741710 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" path="/var/lib/kubelet/pods/04e059a1-1e9b-4a02-a988-b0744036103e/volumes" Nov 03 22:35:37.742375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:37.742351 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" path="/var/lib/kubelet/pods/0ace52e6-3c28-4087-b7c5-81685c6533db/volumes" Nov 03 22:35:37.743225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:37.743200 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" path="/var/lib/kubelet/pods/0e6096f0-23ba-4115-9f27-84fbd9765f1b/volumes" Nov 03 22:35:39.118485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.118452 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h"] Nov 03 22:35:39.119223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119208 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" Nov 03 22:35:39.119267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119224 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" Nov 03 22:35:39.119267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119233 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d96c50f9-d2c2-4acc-8900-562926ab8f18" containerName="revoke-keys" Nov 03 22:35:39.119267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119238 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96c50f9-d2c2-4acc-8900-562926ab8f18" containerName="revoke-keys" Nov 03 22:35:39.119267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119256 4903 cpu_manager.go:414] "containerMap: removing container" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" Nov 03 22:35:39.119267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119261 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" Nov 03 22:35:39.119447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119336 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" Nov 03 22:35:39.119447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119343 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" Nov 03 22:35:39.119447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119356 4903 cpu_manager.go:414] "containerMap: removing container" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" Nov 03 22:35:39.119447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119361 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" Nov 03 22:35:39.119573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119490 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ace52e6-3c28-4087-b7c5-81685c6533db" containerName="istio-proxy" Nov 03 22:35:39.119573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119498 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e059a1-1e9b-4a02-a988-b0744036103e" containerName="istio-proxy" Nov 03 22:35:39.119573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119508 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6096f0-23ba-4115-9f27-84fbd9765f1b" containerName="istio-proxy" Nov 03 22:35:39.119573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119514 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96c50f9-d2c2-4acc-8900-562926ab8f18" containerName="revoke-keys" Nov 03 22:35:39.119573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.119523 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b28bb3-915a-4303-a100-84b16be20ebc" containerName="istio-proxy" Nov 03 22:35:39.125634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.125617 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:39.143973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.143942 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h"] Nov 03 22:35:39.148921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.148898 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4b5e2b\"" Nov 03 22:35:39.152410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.152389 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4b5e2b-dockercfg-26w87\"" Nov 03 22:35:39.203791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.203750 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fb9c\" (UniqueName: \"kubernetes.io/projected/a60a004f-1b91-4d05-a295-3e61f2b88c35-kube-api-access-9fb9c\") pod \"payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h\" (UID: \"a60a004f-1b91-4d05-a295-3e61f2b88c35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:39.204098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.204056 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60a004f-1b91-4d05-a295-3e61f2b88c35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h\" (UID: \"a60a004f-1b91-4d05-a295-3e61f2b88c35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:39.216356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.216274 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw"] Nov 03 22:35:39.268577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.232672 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:39.306627 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.306508 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60a004f-1b91-4d05-a295-3e61f2b88c35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h\" (UID: \"a60a004f-1b91-4d05-a295-3e61f2b88c35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:39.307100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.307000 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tktl\" (UniqueName: \"kubernetes.io/projected/0cad264d-2123-47ad-846d-0f5762bc2b9f-kube-api-access-6tktl\") pod \"payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw\" (UID: \"0cad264d-2123-47ad-846d-0f5762bc2b9f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:39.308492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.307234 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cad264d-2123-47ad-846d-0f5762bc2b9f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw\" (UID: \"0cad264d-2123-47ad-846d-0f5762bc2b9f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:39.308492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.307367 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fb9c\" (UniqueName: \"kubernetes.io/projected/a60a004f-1b91-4d05-a295-3e61f2b88c35-kube-api-access-9fb9c\") pod \"payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h\" (UID: \"a60a004f-1b91-4d05-a295-3e61f2b88c35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:39.308492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.308187 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60a004f-1b91-4d05-a295-3e61f2b88c35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h\" (UID: \"a60a004f-1b91-4d05-a295-3e61f2b88c35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:39.331458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.330275 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw"] Nov 03 22:35:39.372820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.372743 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fb9c\" (UniqueName: \"kubernetes.io/projected/a60a004f-1b91-4d05-a295-3e61f2b88c35-kube-api-access-9fb9c\") pod \"payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h\" (UID: \"a60a004f-1b91-4d05-a295-3e61f2b88c35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:39.410802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.410752 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tktl\" (UniqueName: \"kubernetes.io/projected/0cad264d-2123-47ad-846d-0f5762bc2b9f-kube-api-access-6tktl\") pod \"payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw\" (UID: \"0cad264d-2123-47ad-846d-0f5762bc2b9f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:39.410998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.410825 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cad264d-2123-47ad-846d-0f5762bc2b9f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw\" (UID: \"0cad264d-2123-47ad-846d-0f5762bc2b9f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:39.413788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.413465 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cad264d-2123-47ad-846d-0f5762bc2b9f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw\" (UID: \"0cad264d-2123-47ad-846d-0f5762bc2b9f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:39.434679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.434612 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tktl\" (UniqueName: \"kubernetes.io/projected/0cad264d-2123-47ad-846d-0f5762bc2b9f-kube-api-access-6tktl\") pod \"payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw\" (UID: \"0cad264d-2123-47ad-846d-0f5762bc2b9f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:39.442393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.442356 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:39.615598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.615560 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:39.803977 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:35:39.803945 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60a004f_1b91_4d05_a295_3e61f2b88c35.slice/crio-9c49c6acb6dbdf4d3503650c9a2ee20a791c4a7c25b7a7557e679a0520fc6842 WatchSource:0}: Error finding container 9c49c6acb6dbdf4d3503650c9a2ee20a791c4a7c25b7a7557e679a0520fc6842: Status 404 returned error can't find the container with id 9c49c6acb6dbdf4d3503650c9a2ee20a791c4a7c25b7a7557e679a0520fc6842 Nov 03 22:35:39.807342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.807311 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h"] Nov 03 22:35:39.960115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:39.960080 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw"] Nov 03 22:35:40.613227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:40.613179 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" event={"ID":"0cad264d-2123-47ad-846d-0f5762bc2b9f","Type":"ContainerStarted","Data":"cb8464343bf30b4815c7fdf5947435175a58a82ec8ade42ec0e1edee9379e8e0"} Nov 03 22:35:40.613975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:40.613263 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" event={"ID":"0cad264d-2123-47ad-846d-0f5762bc2b9f","Type":"ContainerStarted","Data":"b7a514154590e38653421d84b04ceb2f17be493d47bfb2432496c3ee25e3eb2b"} Nov 03 22:35:40.613975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:40.613563 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:40.615651 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:40.615622 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" event={"ID":"a60a004f-1b91-4d05-a295-3e61f2b88c35","Type":"ContainerStarted","Data":"dc44bcad55f566541f817847d2775953bc8097503e4c994169a052d8b66bb3b6"} Nov 03 22:35:40.615802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:40.615714 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:40.615802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:40.615742 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" event={"ID":"a60a004f-1b91-4d05-a295-3e61f2b88c35","Type":"ContainerStarted","Data":"9c49c6acb6dbdf4d3503650c9a2ee20a791c4a7c25b7a7557e679a0520fc6842"} Nov 03 22:35:40.659781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:40.659726 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" podStartSLOduration=1.659709095 podStartE2EDuration="1.659709095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:40.657187735 +0000 UTC m=+1367.061128709" watchObservedRunningTime="2026-11-03 22:35:40.659709095 +0000 UTC m=+1367.063650074" Nov 03 22:35:47.329083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.329027 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" podStartSLOduration=8.329013293 podStartE2EDuration="8.329013293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:40.709542845 +0000 UTC m=+1367.113483821" watchObservedRunningTime="2026-11-03 22:35:47.329013293 +0000 UTC m=+1373.732954285" Nov 03 22:35:47.333510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.333483 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99"] Nov 03 22:35:47.338407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.338386 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.358498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.358468 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99"] Nov 03 22:35:47.361528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.361505 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:35:47.370313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.365215 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:35:47.375065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.375039 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:35:47.375264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.375238 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:35:47.456028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.455990 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prgcp\" (UniqueName: \"kubernetes.io/projected/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-kube-api-access-prgcp\") pod \"maas-api-revoke-sub-a0cdebf2-k6s99\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.456237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.456050 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0cdebf2-k6s99\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.456237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.456155 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0cdebf2-k6s99\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.558316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.558261 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prgcp\" (UniqueName: \"kubernetes.io/projected/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-kube-api-access-prgcp\") pod \"maas-api-revoke-sub-a0cdebf2-k6s99\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.558316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.558320 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0cdebf2-k6s99\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.558568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.558371 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0cdebf2-k6s99\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.558974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.558953 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0cdebf2-k6s99\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.560781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.560757 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0cdebf2-k6s99\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.590875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.590810 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prgcp\" (UniqueName: \"kubernetes.io/projected/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-kube-api-access-prgcp\") pod \"maas-api-revoke-sub-a0cdebf2-k6s99\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.656326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.656278 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:47.886311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.886215 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv"] Nov 03 22:35:47.891590 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.891567 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:35:47.909461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.909421 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv"] Nov 03 22:35:47.911855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.911829 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2c0223-dockercfg-fcgm9\"" Nov 03 22:35:47.915022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.914999 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2c0223\"" Nov 03 22:35:47.967070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.967028 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8h52\" (UniqueName: \"kubernetes.io/projected/a3131409-8074-4901-8c15-3966d0c4b147-kube-api-access-v8h52\") pod \"payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv\" (UID: \"a3131409-8074-4901-8c15-3966d0c4b147\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:35:47.967437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.967404 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3131409-8074-4901-8c15-3966d0c4b147-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv\" (UID: \"a3131409-8074-4901-8c15-3966d0c4b147\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:35:47.967437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:47.967410 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq"] Nov 03 22:35:48.060349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.060260 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:35:48.138721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.136000 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8h52\" (UniqueName: \"kubernetes.io/projected/a3131409-8074-4901-8c15-3966d0c4b147-kube-api-access-v8h52\") pod \"payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv\" (UID: \"a3131409-8074-4901-8c15-3966d0c4b147\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:35:48.138721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.136396 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3131409-8074-4901-8c15-3966d0c4b147-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv\" (UID: \"a3131409-8074-4901-8c15-3966d0c4b147\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:35:48.140607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.140526 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3131409-8074-4901-8c15-3966d0c4b147-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv\" (UID: \"a3131409-8074-4901-8c15-3966d0c4b147\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:35:48.246628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.243536 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62grr\" (UniqueName: \"kubernetes.io/projected/4511c67f-6f16-4586-92fc-74a8c4bf8b59-kube-api-access-62grr\") pod \"payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq\" (UID: \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:35:48.247104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.247072 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4511c67f-6f16-4586-92fc-74a8c4bf8b59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq\" (UID: \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:35:48.283675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.283616 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8h52\" (UniqueName: \"kubernetes.io/projected/a3131409-8074-4901-8c15-3966d0c4b147-kube-api-access-v8h52\") pod \"payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv\" (UID: \"a3131409-8074-4901-8c15-3966d0c4b147\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:35:48.285961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.285924 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99"] Nov 03 22:35:48.305419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.305392 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq"] Nov 03 22:35:48.348980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.348951 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62grr\" (UniqueName: \"kubernetes.io/projected/4511c67f-6f16-4586-92fc-74a8c4bf8b59-kube-api-access-62grr\") pod \"payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq\" (UID: \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:35:48.349463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.349423 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4511c67f-6f16-4586-92fc-74a8c4bf8b59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq\" (UID: \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:35:48.350186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.350165 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4511c67f-6f16-4586-92fc-74a8c4bf8b59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq\" (UID: \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:35:48.371048 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.371022 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62grr\" (UniqueName: \"kubernetes.io/projected/4511c67f-6f16-4586-92fc-74a8c4bf8b59-kube-api-access-62grr\") pod \"payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq\" (UID: \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:35:48.388489 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:35:48.388451 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:35:48.508724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.508692 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:35:48.616861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.616831 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:35:48.720603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.720509 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" event={"ID":"ee5ca969-3ba7-47a7-b42c-b17c5a54f893","Type":"ContainerStarted","Data":"42ebb07e50eb0b9b65901380f8506a3c13bfb8bc172df9d2863a97cd27ecdd15"} Nov 03 22:35:48.720603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.720587 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" event={"ID":"ee5ca969-3ba7-47a7-b42c-b17c5a54f893","Type":"ContainerStarted","Data":"ce9b4135b64038d9668065193cfe13508a9cc8f64707eae0dc63fa03e730a164"} Nov 03 22:35:48.808313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.805025 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" podStartSLOduration=1.805005536 podStartE2EDuration="1.805005536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:48.80086023 +0000 UTC m=+1375.204801205" watchObservedRunningTime="2026-11-03 22:35:48.805005536 +0000 UTC m=+1375.208946512" Nov 03 22:35:48.852092 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:35:48.852063 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3131409_8074_4901_8c15_3966d0c4b147.slice/crio-c109c4d5f0f2406a02b89c92d154558dbc26b045dfdaeeef8636b5666497a8a3 WatchSource:0}: Error finding container c109c4d5f0f2406a02b89c92d154558dbc26b045dfdaeeef8636b5666497a8a3: Status 404 returned error can't find the container with id c109c4d5f0f2406a02b89c92d154558dbc26b045dfdaeeef8636b5666497a8a3 Nov 03 22:35:48.868498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.868447 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv"] Nov 03 22:35:48.958925 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:35:48.958895 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4511c67f_6f16_4586_92fc_74a8c4bf8b59.slice/crio-dff5ebbdd69b4b8b8e2587f12a29fe7d3d084ece170c4c9d4d5232b923b21826 WatchSource:0}: Error finding container dff5ebbdd69b4b8b8e2587f12a29fe7d3d084ece170c4c9d4d5232b923b21826: Status 404 returned error can't find the container with id dff5ebbdd69b4b8b8e2587f12a29fe7d3d084ece170c4c9d4d5232b923b21826 Nov 03 22:35:48.961541 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:48.961514 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq"] Nov 03 22:35:49.780237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.780154 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" event={"ID":"4511c67f-6f16-4586-92fc-74a8c4bf8b59","Type":"ContainerStarted","Data":"8486530f7b9de35c956b730b9810c67610f231eda362edce245ec9ec305acde9"} Nov 03 22:35:49.780237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.780226 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" event={"ID":"4511c67f-6f16-4586-92fc-74a8c4bf8b59","Type":"ContainerStarted","Data":"dff5ebbdd69b4b8b8e2587f12a29fe7d3d084ece170c4c9d4d5232b923b21826"} Nov 03 22:35:49.780863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.780413 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:35:49.783334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.783288 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" event={"ID":"a3131409-8074-4901-8c15-3966d0c4b147","Type":"ContainerStarted","Data":"89dbcd66b36f14a6844d8c1d3b507072694873cfff359e913e904a41ec13b420"} Nov 03 22:35:49.783482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.783356 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" event={"ID":"a3131409-8074-4901-8c15-3966d0c4b147","Type":"ContainerStarted","Data":"c109c4d5f0f2406a02b89c92d154558dbc26b045dfdaeeef8636b5666497a8a3"} Nov 03 22:35:49.783482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.783378 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:35:49.785047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.785024 4903 generic.go:356] "Generic (PLEG): container finished" podID="ee5ca969-3ba7-47a7-b42c-b17c5a54f893" containerID="42ebb07e50eb0b9b65901380f8506a3c13bfb8bc172df9d2863a97cd27ecdd15" exitCode=0 Nov 03 22:35:49.785166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.785076 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" event={"ID":"ee5ca969-3ba7-47a7-b42c-b17c5a54f893","Type":"ContainerDied","Data":"42ebb07e50eb0b9b65901380f8506a3c13bfb8bc172df9d2863a97cd27ecdd15"} Nov 03 22:35:49.823544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.823493 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" podStartSLOduration=2.823479082 podStartE2EDuration="2.823479082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:49.817061374 +0000 UTC m=+1376.221002352" watchObservedRunningTime="2026-11-03 22:35:49.823479082 +0000 UTC m=+1376.227420054" Nov 03 22:35:49.897665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:49.897615 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" podStartSLOduration=2.897600364 podStartE2EDuration="2.897600364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:35:49.894637604 +0000 UTC m=+1376.298578581" watchObservedRunningTime="2026-11-03 22:35:49.897600364 +0000 UTC m=+1376.301541338" Nov 03 22:35:51.102087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.102061 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:51.172363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.172324 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-kube-api-access-prgcp\" (UniqueName: \"kubernetes.io/projected/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-kube-api-access-prgcp\") pod \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " Nov 03 22:35:51.172539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.172388 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-xks-ca\") pod \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " Nov 03 22:35:51.172754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.172736 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-openshift-ca\") pod \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\" (UID: \"ee5ca969-3ba7-47a7-b42c-b17c5a54f893\") " Nov 03 22:35:51.173865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.173796 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-openshift-ca" pod "ee5ca969-3ba7-47a7-b42c-b17c5a54f893" (UID: "ee5ca969-3ba7-47a7-b42c-b17c5a54f893"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:35:51.178877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.176521 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-kube-api-access-prgcp" pod "ee5ca969-3ba7-47a7-b42c-b17c5a54f893" (UID: "ee5ca969-3ba7-47a7-b42c-b17c5a54f893"). InnerVolumeSpecName "kube-api-access-prgcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:35:51.179978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.179573 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-xks-ca" pod "ee5ca969-3ba7-47a7-b42c-b17c5a54f893" (UID: "ee5ca969-3ba7-47a7-b42c-b17c5a54f893"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:35:51.275403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.275372 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:51.275403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.275399 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prgcp\" (UniqueName: \"kubernetes.io/projected/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-kube-api-access-prgcp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:51.275403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.275409 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5ca969-3ba7-47a7-b42c-b17c5a54f893-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:35:51.647714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.647686 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:35:51.647888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.647753 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:35:51.805365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.805326 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" event={"ID":"ee5ca969-3ba7-47a7-b42c-b17c5a54f893","Type":"ContainerDied","Data":"ce9b4135b64038d9668065193cfe13508a9cc8f64707eae0dc63fa03e730a164"} Nov 03 22:35:51.805532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.805401 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce9b4135b64038d9668065193cfe13508a9cc8f64707eae0dc63fa03e730a164" Nov 03 22:35:51.805532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:51.805347 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99" Nov 03 22:35:57.156875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:57.156842 4903 scope.go:122] "RemoveContainer" containerID="1031fd06c665c5b664f0e36778f147936aa3ce9c198058ce8ec5c4e05bd0032f" Nov 03 22:35:57.172397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:57.172360 4903 scope.go:122] "RemoveContainer" containerID="f06ddd32e270a94b7c979c6fd05293d7069c89e4fc25018b7d029891f73a5654" Nov 03 22:35:57.199117 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:57.199093 4903 scope.go:122] "RemoveContainer" containerID="7c3000b2aa3967c42459408497a82c6e07163dcbeb8fa440cf35625fe3cf137d" Nov 03 22:35:57.210139 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:57.210117 4903 scope.go:122] "RemoveContainer" containerID="a215eb1a20bcb14a61e40b951e31ed7c91abae9a3557020adb0008ce3e9d99e4" Nov 03 22:35:57.220368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:57.220345 4903 scope.go:122] "RemoveContainer" containerID="431fa2f6a8f2919ac8cd58da46c59fd532260e2a6ca9276e6053b7517a77c5a2" Nov 03 22:35:57.230149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:57.230130 4903 scope.go:122] "RemoveContainer" containerID="2054cff75d0acc5785f8e504972c6305b01738c81d67615a4391c12299e0b304" Nov 03 22:35:57.240288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:57.240269 4903 scope.go:122] "RemoveContainer" containerID="595c249376ec2da3fa0d9fd3738fc0c95303f3825f4945b1b9db4d86d4a6ad04" Nov 03 22:35:57.253531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:35:57.253510 4903 scope.go:122] "RemoveContainer" containerID="04b3620d9b8aec6eaa20573742fdcaf8be21264f810ee527faeb7b11a185a668" Nov 03 22:36:00.797198 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:00.797167 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:36:00.797606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:00.797240 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:36:13.644948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.644870 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7"] Nov 03 22:36:13.645817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.645782 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ee5ca969-3ba7-47a7-b42c-b17c5a54f893" containerName="revoke-keys" Nov 03 22:36:13.645817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.645801 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5ca969-3ba7-47a7-b42c-b17c5a54f893" containerName="revoke-keys" Nov 03 22:36:13.646050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.646032 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5ca969-3ba7-47a7-b42c-b17c5a54f893" containerName="revoke-keys" Nov 03 22:36:13.653265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.653244 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.667438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.667405 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7"] Nov 03 22:36:13.671502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.671475 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:36:13.671679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.671520 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:36:13.696197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.696163 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.696697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.696256 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.696697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.696426 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrrf9\" (UniqueName: \"kubernetes.io/projected/4aa56484-0f48-4d65-a739-389f536a6aa3-kube-api-access-zrrf9\") pod \"maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.741490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.741461 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:36:13.741490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.741470 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:36:13.797518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.797484 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.797731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.797566 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.797731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.797713 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrrf9\" (UniqueName: \"kubernetes.io/projected/4aa56484-0f48-4d65-a739-389f536a6aa3-kube-api-access-zrrf9\") pod \"maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.798238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.798216 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.799938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.799915 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.848604 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.848577 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrrf9\" (UniqueName: \"kubernetes.io/projected/4aa56484-0f48-4d65-a739-389f536a6aa3-kube-api-access-zrrf9\") pod \"maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:13.970588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:13.970558 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:14.291622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:14.291594 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7"] Nov 03 22:36:15.041725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:15.041667 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" event={"ID":"4aa56484-0f48-4d65-a739-389f536a6aa3","Type":"ContainerStarted","Data":"ff18524c5ee666c3f39309daf110a45401d2a81e2da5ac88b857602c84f1122e"} Nov 03 22:36:15.042170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:15.041751 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" event={"ID":"4aa56484-0f48-4d65-a739-389f536a6aa3","Type":"ContainerStarted","Data":"1d0e84df88a073b46ff29954d50477c50b12c6eeae3b6a101079dac4fcf58691"} Nov 03 22:36:15.079495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:15.079438 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" podStartSLOduration=2.07942453 podStartE2EDuration="2.07942453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:15.07849209 +0000 UTC m=+1401.482433065" watchObservedRunningTime="2026-11-03 22:36:15.07942453 +0000 UTC m=+1401.483365499" Nov 03 22:36:15.578937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:15.578905 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8"] Nov 03 22:36:15.579169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:15.579144 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" containerID="cri-o://bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3" gracePeriod=30 Nov 03 22:36:16.053277 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:16.053239 4903 generic.go:356] "Generic (PLEG): container finished" podID="4aa56484-0f48-4d65-a739-389f536a6aa3" containerID="ff18524c5ee666c3f39309daf110a45401d2a81e2da5ac88b857602c84f1122e" exitCode=0 Nov 03 22:36:16.053731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:16.053327 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" event={"ID":"4aa56484-0f48-4d65-a739-389f536a6aa3","Type":"ContainerDied","Data":"ff18524c5ee666c3f39309daf110a45401d2a81e2da5ac88b857602c84f1122e"} Nov 03 22:36:17.261045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.261016 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:17.378201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.378114 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-xks-ca\") pod \"4aa56484-0f48-4d65-a739-389f536a6aa3\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " Nov 03 22:36:17.378397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.378278 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa56484-0f48-4d65-a739-389f536a6aa3-kube-api-access-zrrf9\" (UniqueName: \"kubernetes.io/projected/4aa56484-0f48-4d65-a739-389f536a6aa3-kube-api-access-zrrf9\") pod \"4aa56484-0f48-4d65-a739-389f536a6aa3\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " Nov 03 22:36:17.378585 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.378559 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-openshift-ca\") pod \"4aa56484-0f48-4d65-a739-389f536a6aa3\" (UID: \"4aa56484-0f48-4d65-a739-389f536a6aa3\") " Nov 03 22:36:17.378916 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.378889 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-openshift-ca" pod "4aa56484-0f48-4d65-a739-389f536a6aa3" (UID: "4aa56484-0f48-4d65-a739-389f536a6aa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:17.379350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.379324 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:17.380367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.380339 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-xks-ca" pod "4aa56484-0f48-4d65-a739-389f536a6aa3" (UID: "4aa56484-0f48-4d65-a739-389f536a6aa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:36:17.380540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.380519 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa56484-0f48-4d65-a739-389f536a6aa3-kube-api-access-zrrf9" pod "4aa56484-0f48-4d65-a739-389f536a6aa3" (UID: "4aa56484-0f48-4d65-a739-389f536a6aa3"). InnerVolumeSpecName "kube-api-access-zrrf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:17.481394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.481360 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrrf9\" (UniqueName: \"kubernetes.io/projected/4aa56484-0f48-4d65-a739-389f536a6aa3-kube-api-access-zrrf9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:17.481394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.481390 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa56484-0f48-4d65-a739-389f536a6aa3-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:17.668912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.668878 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w"] Nov 03 22:36:17.669766 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.669748 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4aa56484-0f48-4d65-a739-389f536a6aa3" containerName="revoke-keys" Nov 03 22:36:17.669766 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.669766 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa56484-0f48-4d65-a739-389f536a6aa3" containerName="revoke-keys" Nov 03 22:36:17.669910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.669901 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa56484-0f48-4d65-a739-389f536a6aa3" containerName="revoke-keys" Nov 03 22:36:17.673808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.673789 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:17.687707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.687687 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b826b6-dockercfg-26vvl\"" Nov 03 22:36:17.693024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.692999 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b826b6\"" Nov 03 22:36:17.698212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.698171 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w"] Nov 03 22:36:17.784998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.784959 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw"] Nov 03 22:36:17.795613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.786959 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5wr\" (UniqueName: \"kubernetes.io/projected/91fb5828-1e15-4e28-b558-6b4fd01920a8-kube-api-access-cd5wr\") pod \"payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w\" (UID: \"91fb5828-1e15-4e28-b558-6b4fd01920a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:17.795613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.787014 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fb5828-1e15-4e28-b558-6b4fd01920a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w\" (UID: \"91fb5828-1e15-4e28-b558-6b4fd01920a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:17.807347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.802256 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:17.892096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.889051 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c9b80-60fc-499c-a317-9cdac646d1ab-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw\" (UID: \"535c9b80-60fc-499c-a317-9cdac646d1ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:17.892096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.889387 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6cdw\" (UniqueName: \"kubernetes.io/projected/535c9b80-60fc-499c-a317-9cdac646d1ab-kube-api-access-z6cdw\") pod \"payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw\" (UID: \"535c9b80-60fc-499c-a317-9cdac646d1ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:17.892096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.889470 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5wr\" (UniqueName: \"kubernetes.io/projected/91fb5828-1e15-4e28-b558-6b4fd01920a8-kube-api-access-cd5wr\") pod \"payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w\" (UID: \"91fb5828-1e15-4e28-b558-6b4fd01920a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:17.892096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.889505 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fb5828-1e15-4e28-b558-6b4fd01920a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w\" (UID: \"91fb5828-1e15-4e28-b558-6b4fd01920a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:17.892096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.890534 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fb5828-1e15-4e28-b558-6b4fd01920a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w\" (UID: \"91fb5828-1e15-4e28-b558-6b4fd01920a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:17.912362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.895960 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw"] Nov 03 22:36:17.930609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:17.930458 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5wr\" (UniqueName: \"kubernetes.io/projected/91fb5828-1e15-4e28-b558-6b4fd01920a8-kube-api-access-cd5wr\") pod \"payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w\" (UID: \"91fb5828-1e15-4e28-b558-6b4fd01920a8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:18.021941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.021906 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:18.070850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.022779 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c9b80-60fc-499c-a317-9cdac646d1ab-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw\" (UID: \"535c9b80-60fc-499c-a317-9cdac646d1ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:18.070850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.068979 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6cdw\" (UniqueName: \"kubernetes.io/projected/535c9b80-60fc-499c-a317-9cdac646d1ab-kube-api-access-z6cdw\") pod \"payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw\" (UID: \"535c9b80-60fc-499c-a317-9cdac646d1ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:18.072754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.023593 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c9b80-60fc-499c-a317-9cdac646d1ab-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw\" (UID: \"535c9b80-60fc-499c-a317-9cdac646d1ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:18.093357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.093282 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" event={"ID":"4aa56484-0f48-4d65-a739-389f536a6aa3","Type":"ContainerDied","Data":"1d0e84df88a073b46ff29954d50477c50b12c6eeae3b6a101079dac4fcf58691"} Nov 03 22:36:18.093568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.093551 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d0e84df88a073b46ff29954d50477c50b12c6eeae3b6a101079dac4fcf58691" Nov 03 22:36:18.093634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.093435 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7" Nov 03 22:36:18.103499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.103468 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6cdw\" (UniqueName: \"kubernetes.io/projected/535c9b80-60fc-499c-a317-9cdac646d1ab-kube-api-access-z6cdw\") pod \"payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw\" (UID: \"535c9b80-60fc-499c-a317-9cdac646d1ab\") " pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:18.190131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.189841 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:18.388381 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:36:18.388338 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:36:18.406352 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.405568 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w"] Nov 03 22:36:18.406352 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:36:18.406141 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fb5828_1e15_4e28_b558_6b4fd01920a8.slice/crio-b5c0ddf0344db5c43200b06409db45d20d6df4ccf13a5b55eda4c424ad268d1b WatchSource:0}: Error finding container b5c0ddf0344db5c43200b06409db45d20d6df4ccf13a5b55eda4c424ad268d1b: Status 404 returned error can't find the container with id b5c0ddf0344db5c43200b06409db45d20d6df4ccf13a5b55eda4c424ad268d1b Nov 03 22:36:18.576936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:18.576909 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw"] Nov 03 22:36:19.102991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:19.102950 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" event={"ID":"91fb5828-1e15-4e28-b558-6b4fd01920a8","Type":"ContainerStarted","Data":"b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae"} Nov 03 22:36:19.103167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:19.103016 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" event={"ID":"91fb5828-1e15-4e28-b558-6b4fd01920a8","Type":"ContainerStarted","Data":"b5c0ddf0344db5c43200b06409db45d20d6df4ccf13a5b55eda4c424ad268d1b"} Nov 03 22:36:19.103167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:19.103048 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:19.105267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:19.105221 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" event={"ID":"535c9b80-60fc-499c-a317-9cdac646d1ab","Type":"ContainerStarted","Data":"1715be75365f0f8f396830f16d0f3c58f4403b4de615e4fba8ad9419a0b0345a"} Nov 03 22:36:19.105442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:19.105353 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" event={"ID":"535c9b80-60fc-499c-a317-9cdac646d1ab","Type":"ContainerStarted","Data":"8f2aea74b24cd76ac82ba4e3da135d749fe607cdb0392adc19ce8abe172f7e83"} Nov 03 22:36:19.105442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:19.105389 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:19.147481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:19.147428 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" podStartSLOduration=2.147406686 podStartE2EDuration="2.147406686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:19.144590582 +0000 UTC m=+1405.548531559" watchObservedRunningTime="2026-11-03 22:36:19.147406686 +0000 UTC m=+1405.551347661" Nov 03 22:36:19.191564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:19.191512 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" podStartSLOduration=2.191497866 podStartE2EDuration="2.191497866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:19.187999116 +0000 UTC m=+1405.591940091" watchObservedRunningTime="2026-11-03 22:36:19.191497866 +0000 UTC m=+1405.595438876" Nov 03 22:36:20.866635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:20.866608 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:36:21.020799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.020767 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-token\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-token\") pod \"35eabd1f-e209-4e7a-93e6-8eee2954a256\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " Nov 03 22:36:21.020969 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.020944 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35eabd1f-e209-4e7a-93e6-8eee2954a256-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eabd1f-e209-4e7a-93e6-8eee2954a256-istiod-ca-cert\") pod \"35eabd1f-e209-4e7a-93e6-8eee2954a256\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " Nov 03 22:36:21.021018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021008 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-kube-api-access-lkqg4\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-kube-api-access-lkqg4\") pod \"35eabd1f-e209-4e7a-93e6-8eee2954a256\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " Nov 03 22:36:21.021064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021054 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-data\") pod \"35eabd1f-e209-4e7a-93e6-8eee2954a256\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " Nov 03 22:36:21.021101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021081 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-socket\") pod \"35eabd1f-e209-4e7a-93e6-8eee2954a256\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " Nov 03 22:36:21.021148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021113 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-podinfo\") pod \"35eabd1f-e209-4e7a-93e6-8eee2954a256\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " Nov 03 22:36:21.021148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021139 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-envoy\") pod \"35eabd1f-e209-4e7a-93e6-8eee2954a256\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " Nov 03 22:36:21.021238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021173 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-certs\") pod \"35eabd1f-e209-4e7a-93e6-8eee2954a256\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " Nov 03 22:36:21.021311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021273 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-credential-socket\") pod \"35eabd1f-e209-4e7a-93e6-8eee2954a256\" (UID: \"35eabd1f-e209-4e7a-93e6-8eee2954a256\") " Nov 03 22:36:21.021387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021360 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-socket" pod "35eabd1f-e209-4e7a-93e6-8eee2954a256" (UID: "35eabd1f-e209-4e7a-93e6-8eee2954a256"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:21.021457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021377 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-data" pod "35eabd1f-e209-4e7a-93e6-8eee2954a256" (UID: "35eabd1f-e209-4e7a-93e6-8eee2954a256"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:21.021457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021395 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eabd1f-e209-4e7a-93e6-8eee2954a256-istiod-ca-cert" pod "35eabd1f-e209-4e7a-93e6-8eee2954a256" (UID: "35eabd1f-e209-4e7a-93e6-8eee2954a256"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:21.021647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021622 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-certs" pod "35eabd1f-e209-4e7a-93e6-8eee2954a256" (UID: "35eabd1f-e209-4e7a-93e6-8eee2954a256"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:21.021835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.021811 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-credential-socket" pod "35eabd1f-e209-4e7a-93e6-8eee2954a256" (UID: "35eabd1f-e209-4e7a-93e6-8eee2954a256"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:21.022317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.022271 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eabd1f-e209-4e7a-93e6-8eee2954a256-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:21.022415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.022343 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:21.022415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.022358 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:21.022415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.022372 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:21.022415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.022385 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:21.023348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.023281 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-token" pod "35eabd1f-e209-4e7a-93e6-8eee2954a256" (UID: "35eabd1f-e209-4e7a-93e6-8eee2954a256"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:21.023467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.023435 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-kube-api-access-lkqg4" pod "35eabd1f-e209-4e7a-93e6-8eee2954a256" (UID: "35eabd1f-e209-4e7a-93e6-8eee2954a256"). InnerVolumeSpecName "kube-api-access-lkqg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:21.023716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.023697 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-podinfo" pod "35eabd1f-e209-4e7a-93e6-8eee2954a256" (UID: "35eabd1f-e209-4e7a-93e6-8eee2954a256"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:36:21.023791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.023743 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-envoy" pod "35eabd1f-e209-4e7a-93e6-8eee2954a256" (UID: "35eabd1f-e209-4e7a-93e6-8eee2954a256"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:36:21.123996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.123907 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkqg4\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-kube-api-access-lkqg4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:21.123996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.123937 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:21.123996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.123950 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:21.123996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.123963 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eabd1f-e209-4e7a-93e6-8eee2954a256-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:21.124356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.124188 4903 generic.go:356] "Generic (PLEG): container finished" podID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerID="bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3" exitCode=0 Nov 03 22:36:21.124356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.124244 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" Nov 03 22:36:21.124356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.124269 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" event={"ID":"35eabd1f-e209-4e7a-93e6-8eee2954a256","Type":"ContainerDied","Data":"bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3"} Nov 03 22:36:21.124519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.124358 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8" event={"ID":"35eabd1f-e209-4e7a-93e6-8eee2954a256","Type":"ContainerDied","Data":"f2d34c649fc9b0d2e4fae35348591831693c505fa83fb293940032a67e3cdb8c"} Nov 03 22:36:21.124519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.124383 4903 scope.go:122] "RemoveContainer" containerID="bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3" Nov 03 22:36:21.137016 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.136995 4903 scope.go:122] "RemoveContainer" containerID="bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3" Nov 03 22:36:21.137326 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:36:21.137283 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3\": container with ID starting with bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3 not found: ID does not exist" containerID="bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3" Nov 03 22:36:21.137420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.137334 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3"} err="failed to get container status \"bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3\": rpc error: code = NotFound desc = could not find container \"bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3\": container with ID starting with bb703addd70bd1cdb12cc93f9c2cde22935db22cef7ff7dfc2b1cda4f95a88d3 not found: ID does not exist" Nov 03 22:36:21.174409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.174379 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8"] Nov 03 22:36:21.181861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.181836 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-z7tn8"] Nov 03 22:36:21.738283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:21.738243 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" path="/var/lib/kubelet/pods/35eabd1f-e209-4e7a-93e6-8eee2954a256/volumes" Nov 03 22:36:24.158278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:24.158226 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8"] Nov 03 22:36:29.159622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:29.159575 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" podUID="8c836105-532d-4c37-a4ca-520150b28f7d" containerName="payload-processing" containerID="cri-o://c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f" gracePeriod=25 Nov 03 22:36:29.469433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:29.469405 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:36:29.562016 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:29.561980 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c836105-532d-4c37-a4ca-520150b28f7d-kube-api-access-f2475\" (UniqueName: \"kubernetes.io/projected/8c836105-532d-4c37-a4ca-520150b28f7d-kube-api-access-f2475\") pod \"8c836105-532d-4c37-a4ca-520150b28f7d\" (UID: \"8c836105-532d-4c37-a4ca-520150b28f7d\") " Nov 03 22:36:29.562211 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:29.562059 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c836105-532d-4c37-a4ca-520150b28f7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c836105-532d-4c37-a4ca-520150b28f7d-plugins-config-volume\") pod \"8c836105-532d-4c37-a4ca-520150b28f7d\" (UID: \"8c836105-532d-4c37-a4ca-520150b28f7d\") " Nov 03 22:36:29.562481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:29.562453 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c836105-532d-4c37-a4ca-520150b28f7d-plugins-config-volume" pod "8c836105-532d-4c37-a4ca-520150b28f7d" (UID: "8c836105-532d-4c37-a4ca-520150b28f7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:36:29.562853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:29.562833 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c836105-532d-4c37-a4ca-520150b28f7d-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:29.564970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:29.564943 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c836105-532d-4c37-a4ca-520150b28f7d-kube-api-access-f2475" pod "8c836105-532d-4c37-a4ca-520150b28f7d" (UID: "8c836105-532d-4c37-a4ca-520150b28f7d"). InnerVolumeSpecName "kube-api-access-f2475". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:36:29.665380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:29.665349 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2475\" (UniqueName: \"kubernetes.io/projected/8c836105-532d-4c37-a4ca-520150b28f7d-kube-api-access-f2475\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:36:30.121651 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.121557 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:36:30.121651 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.121634 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:36:30.228974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.228936 4903 generic.go:356] "Generic (PLEG): container finished" podID="8c836105-532d-4c37-a4ca-520150b28f7d" containerID="c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f" exitCode=0 Nov 03 22:36:30.229469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.229010 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" Nov 03 22:36:30.229469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.229043 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" event={"ID":"8c836105-532d-4c37-a4ca-520150b28f7d","Type":"ContainerDied","Data":"c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f"} Nov 03 22:36:30.229469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.229092 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8" event={"ID":"8c836105-532d-4c37-a4ca-520150b28f7d","Type":"ContainerDied","Data":"8f101cbbafb5b3d54de92e6e6b12d768e9fdcc2899790241106b11c3928b7d30"} Nov 03 22:36:30.229469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.229121 4903 scope.go:122] "RemoveContainer" containerID="c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f" Nov 03 22:36:30.235492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.235465 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l"] Nov 03 22:36:30.236870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.236769 4903 cpu_manager.go:414] "containerMap: removing container" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" Nov 03 22:36:30.236870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.236791 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" Nov 03 22:36:30.236870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.236841 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8c836105-532d-4c37-a4ca-520150b28f7d" containerName="payload-processing" Nov 03 22:36:30.236870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.236849 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c836105-532d-4c37-a4ca-520150b28f7d" containerName="payload-processing" Nov 03 22:36:30.237120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.237051 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c836105-532d-4c37-a4ca-520150b28f7d" containerName="payload-processing" Nov 03 22:36:30.237120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.237083 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="35eabd1f-e209-4e7a-93e6-8eee2954a256" containerName="istio-proxy" Nov 03 22:36:30.244751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.244728 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.246587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.246563 4903 scope.go:122] "RemoveContainer" containerID="c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f" Nov 03 22:36:30.246965 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:36:30.246935 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f\": container with ID starting with c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f not found: ID does not exist" containerID="c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f" Nov 03 22:36:30.247057 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.246974 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f"} err="failed to get container status \"c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f\": rpc error: code = NotFound desc = could not find container \"c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f\": container with ID starting with c798627f4fe26b053c24d9446f53644486988abcb58b2b535b16534ae5434f2f not found: ID does not exist" Nov 03 22:36:30.263542 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.263516 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l"] Nov 03 22:36:30.278500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.278466 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5dd3b1-openshift-default-dockercfg-g59wc\"" Nov 03 22:36:30.354449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.354377 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8"] Nov 03 22:36:30.369539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.369512 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c5345-6df87f7798-fkts8"] Nov 03 22:36:30.381552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.381486 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-socket\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.381552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.381540 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-data\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.381702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.381655 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbae0ea0-7a75-4080-96b9-aca308be40ec-istiod-ca-cert\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.381702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.381697 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-certs\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.381794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.381713 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-token\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.381864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.381836 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-envoy\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.381913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.381891 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ctg9\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-kube-api-access-5ctg9\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.381949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.381943 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-credential-socket\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.382006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.381994 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-podinfo\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.489448 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-credential-socket\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.489549 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-podinfo\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.489710 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-socket\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.489765 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-data\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.489877 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbae0ea0-7a75-4080-96b9-aca308be40ec-istiod-ca-cert\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.489903 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-credential-socket\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.489945 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-certs\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.489971 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-token\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.490057 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-envoy\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.490105 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ctg9\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-kube-api-access-5ctg9\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.490319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.490170 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-socket\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.491055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.490529 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-data\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.491198 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.491173 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbae0ea0-7a75-4080-96b9-aca308be40ec-istiod-ca-cert\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.491259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.491201 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-certs\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.513946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.513879 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-podinfo\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.521319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.515590 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-envoy\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.533320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.531435 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-token\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.557315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.553265 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ctg9\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-kube-api-access-5ctg9\") pod \"e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.585049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.585017 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:30.989600 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:36:30.989573 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbae0ea0_7a75_4080_96b9_aca308be40ec.slice/crio-6c7cedc9b2787247566bebae00c4b4421a304fad51b6e36111fd11827a1b74a9 WatchSource:0}: Error finding container 6c7cedc9b2787247566bebae00c4b4421a304fad51b6e36111fd11827a1b74a9: Status 404 returned error can't find the container with id 6c7cedc9b2787247566bebae00c4b4421a304fad51b6e36111fd11827a1b74a9 Nov 03 22:36:30.990927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.990901 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l"] Nov 03 22:36:30.993105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.993072 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:36:30.993223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.993138 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:36:30.993223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:30.993166 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:36:31.240303 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:31.240210 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" event={"ID":"bbae0ea0-7a75-4080-96b9-aca308be40ec","Type":"ContainerStarted","Data":"a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8"} Nov 03 22:36:31.240303 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:31.240261 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" event={"ID":"bbae0ea0-7a75-4080-96b9-aca308be40ec","Type":"ContainerStarted","Data":"6c7cedc9b2787247566bebae00c4b4421a304fad51b6e36111fd11827a1b74a9"} Nov 03 22:36:31.302335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:31.302239 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" podStartSLOduration=1.302225914 podStartE2EDuration="1.302225914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:31.301452074 +0000 UTC m=+1417.705393049" watchObservedRunningTime="2026-11-03 22:36:31.302225914 +0000 UTC m=+1417.706166889" Nov 03 22:36:31.586456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:31.586380 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:31.740068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:31.740034 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c836105-532d-4c37-a4ca-520150b28f7d" path="/var/lib/kubelet/pods/8c836105-532d-4c37-a4ca-520150b28f7d/volumes" Nov 03 22:36:32.557265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:32.557233 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc"] Nov 03 22:36:32.568218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:32.568186 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-kvrjc"] Nov 03 22:36:32.609145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:32.609108 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:33.297871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:33.297844 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:33.299082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:33.299057 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:36:33.738834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:33.738796 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96c50f9-d2c2-4acc-8900-562926ab8f18" path="/var/lib/kubelet/pods/d96c50f9-d2c2-4acc-8900-562926ab8f18/volumes" Nov 03 22:36:35.660528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:35.660494 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6"] Nov 03 22:36:35.666828 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:35.666808 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:35.685745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:35.685724 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6"] Nov 03 22:36:35.690438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:35.690399 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5dd3b1\"" Nov 03 22:36:35.690684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:35.690658 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-5dd3b1-dockercfg-rlwm6\"" Nov 03 22:36:35.964436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:35.948199 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4912538-fee8-4c7e-8b44-fff393346bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6\" (UID: \"a4912538-fee8-4c7e-8b44-fff393346bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:35.964436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:35.948271 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5znr\" (UniqueName: \"kubernetes.io/projected/a4912538-fee8-4c7e-8b44-fff393346bf0-kube-api-access-x5znr\") pod \"payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6\" (UID: \"a4912538-fee8-4c7e-8b44-fff393346bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:35.995596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:35.995562 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj"] Nov 03 22:36:36.004179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.004149 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj"] Nov 03 22:36:36.004531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.004508 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:36.049799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.049758 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4912538-fee8-4c7e-8b44-fff393346bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6\" (UID: \"a4912538-fee8-4c7e-8b44-fff393346bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:36.049971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.049819 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5znr\" (UniqueName: \"kubernetes.io/projected/a4912538-fee8-4c7e-8b44-fff393346bf0-kube-api-access-x5znr\") pod \"payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6\" (UID: \"a4912538-fee8-4c7e-8b44-fff393346bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:36.050776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.050744 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4912538-fee8-4c7e-8b44-fff393346bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6\" (UID: \"a4912538-fee8-4c7e-8b44-fff393346bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:36.073613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.073581 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5znr\" (UniqueName: \"kubernetes.io/projected/a4912538-fee8-4c7e-8b44-fff393346bf0-kube-api-access-x5znr\") pod \"payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6\" (UID: \"a4912538-fee8-4c7e-8b44-fff393346bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:36.151390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.151337 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj\" (UID: \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:36.151671 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.151652 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzkcr\" (UniqueName: \"kubernetes.io/projected/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-kube-api-access-qzkcr\") pod \"payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj\" (UID: \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:36.253715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.253638 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzkcr\" (UniqueName: \"kubernetes.io/projected/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-kube-api-access-qzkcr\") pod \"payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj\" (UID: \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:36.253886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.253790 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj\" (UID: \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:36.254546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.254520 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj\" (UID: \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:36.278086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.278060 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzkcr\" (UniqueName: \"kubernetes.io/projected/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-kube-api-access-qzkcr\") pod \"payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj\" (UID: \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\") " pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:36.286743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.286721 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:36.322004 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.321954 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:36.600624 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:36:36.600595 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4912538_fee8_4c7e_8b44_fff393346bf0.slice/crio-314ff03a94babf0c6630f74c4271f697b695891c513ed2d504a6615aa8002d59 WatchSource:0}: Error finding container 314ff03a94babf0c6630f74c4271f697b695891c513ed2d504a6615aa8002d59: Status 404 returned error can't find the container with id 314ff03a94babf0c6630f74c4271f697b695891c513ed2d504a6615aa8002d59 Nov 03 22:36:36.601696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.601668 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6"] Nov 03 22:36:36.649909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:36.649886 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj"] Nov 03 22:36:36.659304 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:36:36.658187 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2889ddb8_cf2b_4d0a_bb3a_49447d395d2b.slice/crio-bf2740208e2816b8757581cfe6c7ea768300b3b965bc3db898ade74953273a1e WatchSource:0}: Error finding container bf2740208e2816b8757581cfe6c7ea768300b3b965bc3db898ade74953273a1e: Status 404 returned error can't find the container with id bf2740208e2816b8757581cfe6c7ea768300b3b965bc3db898ade74953273a1e Nov 03 22:36:37.333241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:37.333205 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" event={"ID":"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b","Type":"ContainerStarted","Data":"22776e5f336c169d0a60dec5a0adfd4c3f64adba86d486b4576f2fa097e1949d"} Nov 03 22:36:37.333706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:37.333275 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:37.333706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:37.333313 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" event={"ID":"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b","Type":"ContainerStarted","Data":"bf2740208e2816b8757581cfe6c7ea768300b3b965bc3db898ade74953273a1e"} Nov 03 22:36:37.335043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:37.335016 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" event={"ID":"a4912538-fee8-4c7e-8b44-fff393346bf0","Type":"ContainerStarted","Data":"45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce"} Nov 03 22:36:37.335156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:37.335059 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" event={"ID":"a4912538-fee8-4c7e-8b44-fff393346bf0","Type":"ContainerStarted","Data":"314ff03a94babf0c6630f74c4271f697b695891c513ed2d504a6615aa8002d59"} Nov 03 22:36:37.335156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:37.335100 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:37.374783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:37.374718 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" podStartSLOduration=2.374703749 podStartE2EDuration="2.374703749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:37.369687209 +0000 UTC m=+1423.773628188" watchObservedRunningTime="2026-11-03 22:36:37.374703749 +0000 UTC m=+1423.778644725" Nov 03 22:36:37.412219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:37.412161 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" podStartSLOduration=2.41214332 podStartE2EDuration="2.41214332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:37.411200318 +0000 UTC m=+1423.815141296" watchObservedRunningTime="2026-11-03 22:36:37.41214332 +0000 UTC m=+1423.816084299" Nov 03 22:36:37.921926 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:37.921882 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:36:41.087020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.086982 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct"] Nov 03 22:36:41.094501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.094476 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:41.122715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.122687 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8b1efeb6-dockercfg-dzfkg\"" Nov 03 22:36:41.128314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.126421 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct"] Nov 03 22:36:41.159396 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.159366 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz"] Nov 03 22:36:41.176606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.176574 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:41.180327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.178768 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8b1efeb6\"" Nov 03 22:36:41.180327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.179702 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz"] Nov 03 22:36:41.210596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.210550 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct\" (UID: \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:41.210789 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.210754 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjhf\" (UniqueName: \"kubernetes.io/projected/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-kube-api-access-xcjhf\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct\" (UID: \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:41.405869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.364598 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct\" (UID: \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:41.405869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.364801 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjhf\" (UniqueName: \"kubernetes.io/projected/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-kube-api-access-xcjhf\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct\" (UID: \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:41.405869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.366390 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct\" (UID: \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:41.433030 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.427197 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjhf\" (UniqueName: \"kubernetes.io/projected/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-kube-api-access-xcjhf\") pod \"payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct\" (UID: \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\") " pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:41.474518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.473169 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbr92\" (UniqueName: \"kubernetes.io/projected/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-kube-api-access-hbr92\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz\" (UID: \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:41.474518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.473313 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz\" (UID: \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:41.574560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.574523 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbr92\" (UniqueName: \"kubernetes.io/projected/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-kube-api-access-hbr92\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz\" (UID: \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:41.574783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.574628 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz\" (UID: \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:41.575459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.575435 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz\" (UID: \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:41.596491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.596455 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbr92\" (UniqueName: \"kubernetes.io/projected/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-kube-api-access-hbr92\") pod \"payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz\" (UID: \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:41.683021 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.682925 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:41.716721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:41.716508 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:42.026956 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:36:42.026926 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152ffdb2_bb93_4293_a4ae_0ec17515b0d6.slice/crio-75a52bfe80206af2329ab198f3d6e1bc4c0d0b521730de4316a273f6c13dd02d WatchSource:0}: Error finding container 75a52bfe80206af2329ab198f3d6e1bc4c0d0b521730de4316a273f6c13dd02d: Status 404 returned error can't find the container with id 75a52bfe80206af2329ab198f3d6e1bc4c0d0b521730de4316a273f6c13dd02d Nov 03 22:36:42.027937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.027912 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz"] Nov 03 22:36:42.085828 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:36:42.085793 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5d8452_9a8e_4422_816c_e5ab42caa1bb.slice/crio-c625183428b44cd71c33d2030161ca72093e4e9869eb6670f7f01062a1f2de77 WatchSource:0}: Error finding container c625183428b44cd71c33d2030161ca72093e4e9869eb6670f7f01062a1f2de77: Status 404 returned error can't find the container with id c625183428b44cd71c33d2030161ca72093e4e9869eb6670f7f01062a1f2de77 Nov 03 22:36:42.086875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.086848 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct"] Nov 03 22:36:42.477382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.477346 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" event={"ID":"8d5d8452-9a8e-4422-816c-e5ab42caa1bb","Type":"ContainerStarted","Data":"c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c"} Nov 03 22:36:42.477814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.477404 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" event={"ID":"8d5d8452-9a8e-4422-816c-e5ab42caa1bb","Type":"ContainerStarted","Data":"c625183428b44cd71c33d2030161ca72093e4e9869eb6670f7f01062a1f2de77"} Nov 03 22:36:42.477814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.477433 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:42.479467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.479440 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" event={"ID":"152ffdb2-bb93-4293-a4ae-0ec17515b0d6","Type":"ContainerStarted","Data":"df38ef3bf3292b61be3d59590aa04108d2801ff674c500f40f63a271581f011b"} Nov 03 22:36:42.479582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.479484 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" event={"ID":"152ffdb2-bb93-4293-a4ae-0ec17515b0d6","Type":"ContainerStarted","Data":"75a52bfe80206af2329ab198f3d6e1bc4c0d0b521730de4316a273f6c13dd02d"} Nov 03 22:36:42.479582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.479505 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:42.522006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.521949 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" podStartSLOduration=1.521935197 podStartE2EDuration="1.521935197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:42.517979559 +0000 UTC m=+1428.921920535" watchObservedRunningTime="2026-11-03 22:36:42.521935197 +0000 UTC m=+1428.925876172" Nov 03 22:36:42.563361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:42.563285 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" podStartSLOduration=1.563271627 podStartE2EDuration="1.563271627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:36:42.562097941 +0000 UTC m=+1428.966038916" watchObservedRunningTime="2026-11-03 22:36:42.563271627 +0000 UTC m=+1428.967212634" Nov 03 22:36:48.348608 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:48.348578 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:36:48.349085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:48.348648 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:36:48.385696 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:36:48.385650 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:36:53.528317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:53.524486 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:36:53.542316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:53.540473 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:36:57.395453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:57.395410 4903 scope.go:122] "RemoveContainer" containerID="a7d08bc9a5875f293fe6f6a2f84c32645b3da9795a50e18544d1e8d5ac9377f1" Nov 03 22:36:57.409042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:57.409016 4903 scope.go:122] "RemoveContainer" containerID="f5d97d89fcb917ea9e8b0eeeb8f2b0d20c087cef6217422c7040120e231a721f" Nov 03 22:36:57.424464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:36:57.424436 4903 scope.go:122] "RemoveContainer" containerID="53a564f5c975d16b113032ee57634f3ea0e25363d78397e96259648a113cf32a" Nov 03 22:37:06.044130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.044100 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6"] Nov 03 22:37:06.049338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.049315 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:06.059586 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.059560 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9"] Nov 03 22:37:06.063307 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.062964 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2d6076\"" Nov 03 22:37:06.073005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.072973 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6"] Nov 03 22:37:06.079640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.079612 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a7419de-nzkr9"] Nov 03 22:37:06.109816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.109780 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f562a97d-3a7d-476a-a5c6-9b102811d26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6\" (UID: \"f562a97d-3a7d-476a-a5c6-9b102811d26b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:06.110044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.110027 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmqjx\" (UniqueName: \"kubernetes.io/projected/f562a97d-3a7d-476a-a5c6-9b102811d26b-kube-api-access-jmqjx\") pod \"payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6\" (UID: \"f562a97d-3a7d-476a-a5c6-9b102811d26b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:06.219657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.219598 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f562a97d-3a7d-476a-a5c6-9b102811d26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6\" (UID: \"f562a97d-3a7d-476a-a5c6-9b102811d26b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:06.219875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.219857 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmqjx\" (UniqueName: \"kubernetes.io/projected/f562a97d-3a7d-476a-a5c6-9b102811d26b-kube-api-access-jmqjx\") pod \"payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6\" (UID: \"f562a97d-3a7d-476a-a5c6-9b102811d26b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:06.221288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.221258 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f562a97d-3a7d-476a-a5c6-9b102811d26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6\" (UID: \"f562a97d-3a7d-476a-a5c6-9b102811d26b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:06.296386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.290838 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmqjx\" (UniqueName: \"kubernetes.io/projected/f562a97d-3a7d-476a-a5c6-9b102811d26b-kube-api-access-jmqjx\") pod \"payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6\" (UID: \"f562a97d-3a7d-476a-a5c6-9b102811d26b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:06.375278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.373831 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:06.757834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.757803 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6"] Nov 03 22:37:06.758305 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:37:06.758266 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf562a97d_3a7d_476a_a5c6_9b102811d26b.slice/crio-9f1d1cc0494e13fa43e30e93f6f5c33873bfeaeea358d2204bc76da5b28c8fbd WatchSource:0}: Error finding container 9f1d1cc0494e13fa43e30e93f6f5c33873bfeaeea358d2204bc76da5b28c8fbd: Status 404 returned error can't find the container with id 9f1d1cc0494e13fa43e30e93f6f5c33873bfeaeea358d2204bc76da5b28c8fbd Nov 03 22:37:06.869242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.869190 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" event={"ID":"f562a97d-3a7d-476a-a5c6-9b102811d26b","Type":"ContainerStarted","Data":"2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9"} Nov 03 22:37:06.869434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.869353 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:06.869434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.869399 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" event={"ID":"f562a97d-3a7d-476a-a5c6-9b102811d26b","Type":"ContainerStarted","Data":"9f1d1cc0494e13fa43e30e93f6f5c33873bfeaeea358d2204bc76da5b28c8fbd"} Nov 03 22:37:06.906936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:06.906883 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" podStartSLOduration=0.906871329 podStartE2EDuration="906.871329ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:06.906197912 +0000 UTC m=+1453.310138885" watchObservedRunningTime="2026-11-03 22:37:06.906871329 +0000 UTC m=+1453.310812304" Nov 03 22:37:07.743684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:07.743648 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd935e3-a795-44e5-a570-39a00ef9bbaa" path="/var/lib/kubelet/pods/cdd935e3-a795-44e5-a570-39a00ef9bbaa/volumes" Nov 03 22:37:09.982427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:09.982392 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8"] Nov 03 22:37:09.982911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:09.982594 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" containerID="cri-o://55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e" gracePeriod=30 Nov 03 22:37:11.044174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:11.044137 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d"] Nov 03 22:37:11.056578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:11.056545 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff769914-svw6d"] Nov 03 22:37:11.795487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:11.795454 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53abd825-5969-4f59-9119-9b0abdc7d74b" path="/var/lib/kubelet/pods/53abd825-5969-4f59-9119-9b0abdc7d74b/volumes" Nov 03 22:37:15.331640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.331608 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:37:15.423684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.423647 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-token\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-token\") pod \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " Nov 03 22:37:15.423961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.423852 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-certs\") pod \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " Nov 03 22:37:15.424208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.424181 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-certs" pod "98d3f93d-025c-4aa4-88f1-af33c03fbb13" (UID: "98d3f93d-025c-4aa4-88f1-af33c03fbb13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:15.424347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.424273 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-kube-api-access-xjsmg\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-kube-api-access-xjsmg\") pod \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " Nov 03 22:37:15.424416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.424380 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istiod-ca-cert\") pod \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " Nov 03 22:37:15.424871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.424761 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istiod-ca-cert" pod "98d3f93d-025c-4aa4-88f1-af33c03fbb13" (UID: "98d3f93d-025c-4aa4-88f1-af33c03fbb13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:15.424979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.424880 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-data\") pod \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " Nov 03 22:37:15.424979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.424940 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-socket\") pod \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " Nov 03 22:37:15.425107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.425081 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-credential-socket\") pod \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " Nov 03 22:37:15.425166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.425129 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-podinfo\") pod \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " Nov 03 22:37:15.425225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.425164 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-envoy\") pod \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\" (UID: \"98d3f93d-025c-4aa4-88f1-af33c03fbb13\") " Nov 03 22:37:15.425371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.424846 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-data" pod "98d3f93d-025c-4aa4-88f1-af33c03fbb13" (UID: "98d3f93d-025c-4aa4-88f1-af33c03fbb13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:15.425672 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.425651 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-credential-socket" pod "98d3f93d-025c-4aa4-88f1-af33c03fbb13" (UID: "98d3f93d-025c-4aa4-88f1-af33c03fbb13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:15.426198 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.426176 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:15.426346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.426331 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:15.426467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.426454 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:15.426575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.426562 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:15.426707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.426502 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-token" pod "98d3f93d-025c-4aa4-88f1-af33c03fbb13" (UID: "98d3f93d-025c-4aa4-88f1-af33c03fbb13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:15.426817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.426720 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-socket" pod "98d3f93d-025c-4aa4-88f1-af33c03fbb13" (UID: "98d3f93d-025c-4aa4-88f1-af33c03fbb13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:15.426817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.426720 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-kube-api-access-xjsmg" pod "98d3f93d-025c-4aa4-88f1-af33c03fbb13" (UID: "98d3f93d-025c-4aa4-88f1-af33c03fbb13"). InnerVolumeSpecName "kube-api-access-xjsmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:15.429138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.429110 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-envoy" pod "98d3f93d-025c-4aa4-88f1-af33c03fbb13" (UID: "98d3f93d-025c-4aa4-88f1-af33c03fbb13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:37:15.429776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.429752 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-podinfo" pod "98d3f93d-025c-4aa4-88f1-af33c03fbb13" (UID: "98d3f93d-025c-4aa4-88f1-af33c03fbb13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:37:15.528202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.528113 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjsmg\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-kube-api-access-xjsmg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:15.528202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.528146 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:15.528202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.528156 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:15.528202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.528165 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:15.528202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.528173 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d3f93d-025c-4aa4-88f1-af33c03fbb13-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:15.923354 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.923315 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2"] Nov 03 22:37:15.924192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.924175 4903 cpu_manager.go:414] "containerMap: removing container" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" Nov 03 22:37:15.924249 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.924194 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" Nov 03 22:37:15.924379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.924369 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerName="istio-proxy" Nov 03 22:37:15.928899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.928875 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:15.943648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.943620 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-22edc1\"" Nov 03 22:37:15.948934 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.948908 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2"] Nov 03 22:37:15.987622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.987590 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47"] Nov 03 22:37:15.994016 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:15.993992 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:16.011551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.011497 4903 generic.go:356] "Generic (PLEG): container finished" podID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" containerID="55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e" exitCode=0 Nov 03 22:37:16.011706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.011582 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" Nov 03 22:37:16.011782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.011758 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" event={"ID":"98d3f93d-025c-4aa4-88f1-af33c03fbb13","Type":"ContainerDied","Data":"55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e"} Nov 03 22:37:16.011836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.011818 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8" event={"ID":"98d3f93d-025c-4aa4-88f1-af33c03fbb13","Type":"ContainerDied","Data":"54fc8f586281e1bd25859ec8a58baab2fa706de5c660a699fc227b4043d0c259"} Nov 03 22:37:16.011899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.011849 4903 scope.go:122] "RemoveContainer" containerID="55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e" Nov 03 22:37:16.012951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.012928 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47"] Nov 03 22:37:16.013151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.013137 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-22edc1-dockercfg-df7ph\"" Nov 03 22:37:16.027158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.027134 4903 scope.go:122] "RemoveContainer" containerID="55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e" Nov 03 22:37:16.028121 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:37:16.028096 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e\": container with ID starting with 55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e not found: ID does not exist" containerID="55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e" Nov 03 22:37:16.028234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.028126 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e"} err="failed to get container status \"55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e\": rpc error: code = NotFound desc = could not find container \"55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e\": container with ID starting with 55ed0407a02985793dad0146c44dcf756ec98913a3a4e6774b925ff138f1f68e not found: ID does not exist" Nov 03 22:37:16.043763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.039259 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-plugins-config-volume\") pod \"payload-processing-e2e-api-b-22edc1-998976c8b-4whv2\" (UID: \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\") " pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:16.043763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.039521 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f827w\" (UniqueName: \"kubernetes.io/projected/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-kube-api-access-f827w\") pod \"payload-processing-e2e-api-b-22edc1-998976c8b-4whv2\" (UID: \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\") " pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:16.184377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.175197 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5rvr\" (UniqueName: \"kubernetes.io/projected/07596a42-3f87-4a8c-9362-d78b4cb27e62-kube-api-access-l5rvr\") pod \"payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47\" (UID: \"07596a42-3f87-4a8c-9362-d78b4cb27e62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:16.184377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.175254 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f827w\" (UniqueName: \"kubernetes.io/projected/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-kube-api-access-f827w\") pod \"payload-processing-e2e-api-b-22edc1-998976c8b-4whv2\" (UID: \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\") " pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:16.184377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.175353 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07596a42-3f87-4a8c-9362-d78b4cb27e62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47\" (UID: \"07596a42-3f87-4a8c-9362-d78b4cb27e62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:16.184377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.175521 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-plugins-config-volume\") pod \"payload-processing-e2e-api-b-22edc1-998976c8b-4whv2\" (UID: \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\") " pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:16.220318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.213273 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-plugins-config-volume\") pod \"payload-processing-e2e-api-b-22edc1-998976c8b-4whv2\" (UID: \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\") " pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:16.220318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.219042 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8"] Nov 03 22:37:16.244946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.241003 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f827w\" (UniqueName: \"kubernetes.io/projected/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-kube-api-access-f827w\") pod \"payload-processing-e2e-api-b-22edc1-998976c8b-4whv2\" (UID: \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\") " pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:16.278714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.277974 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5rvr\" (UniqueName: \"kubernetes.io/projected/07596a42-3f87-4a8c-9362-d78b4cb27e62-kube-api-access-l5rvr\") pod \"payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47\" (UID: \"07596a42-3f87-4a8c-9362-d78b4cb27e62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:16.278714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.278078 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07596a42-3f87-4a8c-9362-d78b4cb27e62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47\" (UID: \"07596a42-3f87-4a8c-9362-d78b4cb27e62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:16.284311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.279079 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07596a42-3f87-4a8c-9362-d78b4cb27e62-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47\" (UID: \"07596a42-3f87-4a8c-9362-d78b4cb27e62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:16.284311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.279252 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:16.300838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.300777 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-mcjd8"] Nov 03 22:37:16.310217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.310159 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5rvr\" (UniqueName: \"kubernetes.io/projected/07596a42-3f87-4a8c-9362-d78b4cb27e62-kube-api-access-l5rvr\") pod \"payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47\" (UID: \"07596a42-3f87-4a8c-9362-d78b4cb27e62\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:16.315456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.315429 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:16.625343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.625316 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2"] Nov 03 22:37:16.625906 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:37:16.625442 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81d958b_c19c_4f8c_97f4_4d6a5a46a921.slice/crio-59ff0eec42110b93649b42950dc12a751da00a929647080abd3a769d64a9f84e WatchSource:0}: Error finding container 59ff0eec42110b93649b42950dc12a751da00a929647080abd3a769d64a9f84e: Status 404 returned error can't find the container with id 59ff0eec42110b93649b42950dc12a751da00a929647080abd3a769d64a9f84e Nov 03 22:37:16.656048 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:37:16.656002 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07596a42_3f87_4a8c_9362_d78b4cb27e62.slice/crio-4181afd0870c59584b3472d9e5faea641423af6a7ffeb5ac725b014cd4cd3ee1 WatchSource:0}: Error finding container 4181afd0870c59584b3472d9e5faea641423af6a7ffeb5ac725b014cd4cd3ee1: Status 404 returned error can't find the container with id 4181afd0870c59584b3472d9e5faea641423af6a7ffeb5ac725b014cd4cd3ee1 Nov 03 22:37:16.658490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:16.658176 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47"] Nov 03 22:37:17.022071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.022029 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" event={"ID":"f81d958b-c19c-4f8c-97f4-4d6a5a46a921","Type":"ContainerStarted","Data":"6157ce885155799b59e483136347e9e0c510201968033d058f032462b05dfe5d"} Nov 03 22:37:17.022257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.022111 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" event={"ID":"f81d958b-c19c-4f8c-97f4-4d6a5a46a921","Type":"ContainerStarted","Data":"59ff0eec42110b93649b42950dc12a751da00a929647080abd3a769d64a9f84e"} Nov 03 22:37:17.022257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.022142 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:17.024978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.024949 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" event={"ID":"07596a42-3f87-4a8c-9362-d78b4cb27e62","Type":"ContainerStarted","Data":"b707aa2f2ef8ffe1111479101871f1f459623d47cdef5b31e04eb4e2d60b2e10"} Nov 03 22:37:17.025121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.024998 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" event={"ID":"07596a42-3f87-4a8c-9362-d78b4cb27e62","Type":"ContainerStarted","Data":"4181afd0870c59584b3472d9e5faea641423af6a7ffeb5ac725b014cd4cd3ee1"} Nov 03 22:37:17.025121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.025068 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:17.185423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.185366 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" podStartSLOduration=2.185350873 podStartE2EDuration="2.185350873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:17.180185997 +0000 UTC m=+1463.584126972" watchObservedRunningTime="2026-11-03 22:37:17.185350873 +0000 UTC m=+1463.589291881" Nov 03 22:37:17.270999 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.270948 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" podStartSLOduration=2.270933504 podStartE2EDuration="2.270933504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:17.265947686 +0000 UTC m=+1463.669888660" watchObservedRunningTime="2026-11-03 22:37:17.270933504 +0000 UTC m=+1463.674874501" Nov 03 22:37:17.738164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.738123 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d3f93d-025c-4aa4-88f1-af33c03fbb13" path="/var/lib/kubelet/pods/98d3f93d-025c-4aa4-88f1-af33c03fbb13/volumes" Nov 03 22:37:17.882506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:17.882477 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:37:18.385026 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:37:18.384989 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:37:28.040333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:28.038275 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:37:28.040333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:28.038406 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:37:28.740381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:28.740339 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7"] Nov 03 22:37:28.752793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:28.752755 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c5345-rhrl7"] Nov 03 22:37:29.738410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:29.738374 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa56484-0f48-4d65-a739-389f536a6aa3" path="/var/lib/kubelet/pods/4aa56484-0f48-4d65-a739-389f536a6aa3/volumes" Nov 03 22:37:37.045879 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.045793 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv"] Nov 03 22:37:37.052982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.052958 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.068941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.068914 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:37:37.086147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.086115 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:37.086374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.086223 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:37:37.087203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.087177 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv"] Nov 03 22:37:37.101333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.098823 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.101333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.098931 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvgc\" (UniqueName: \"kubernetes.io/projected/0e3852f9-c306-4bb0-862f-cbc9914f1daf-kube-api-access-7zvgc\") pod \"maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.101333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.098970 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.182308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.182264 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:37:37.200301 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.200258 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.200486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.200378 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvgc\" (UniqueName: \"kubernetes.io/projected/0e3852f9-c306-4bb0-862f-cbc9914f1daf-kube-api-access-7zvgc\") pod \"maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.200486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.200417 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.200960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.200938 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.202953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.202928 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.232125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.232091 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvgc\" (UniqueName: \"kubernetes.io/projected/0e3852f9-c306-4bb0-862f-cbc9914f1daf-kube-api-access-7zvgc\") pod \"maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.372006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.371920 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:37.700997 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:37:37.700959 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3852f9_c306_4bb0_862f_cbc9914f1daf.slice/crio-ab36b44898eb3ab819bde0bbcbff0b038e35bda9925a771cc44cda614562ff25 WatchSource:0}: Error finding container ab36b44898eb3ab819bde0bbcbff0b038e35bda9925a771cc44cda614562ff25: Status 404 returned error can't find the container with id ab36b44898eb3ab819bde0bbcbff0b038e35bda9925a771cc44cda614562ff25 Nov 03 22:37:37.701805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:37.701779 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv"] Nov 03 22:37:38.231801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:38.231717 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" event={"ID":"0e3852f9-c306-4bb0-862f-cbc9914f1daf","Type":"ContainerStarted","Data":"f7f9a444966779560bd7e61f9223e1bf4150cfec88bce66e73a12f404cb378fb"} Nov 03 22:37:38.231801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:38.231783 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" event={"ID":"0e3852f9-c306-4bb0-862f-cbc9914f1daf","Type":"ContainerStarted","Data":"ab36b44898eb3ab819bde0bbcbff0b038e35bda9925a771cc44cda614562ff25"} Nov 03 22:37:38.270729 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:38.270677 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" podStartSLOduration=1.27066432 podStartE2EDuration="1.27066432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:38.267617646 +0000 UTC m=+1484.671558621" watchObservedRunningTime="2026-11-03 22:37:38.27066432 +0000 UTC m=+1484.674605295" Nov 03 22:37:39.242172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:39.242139 4903 generic.go:356] "Generic (PLEG): container finished" podID="0e3852f9-c306-4bb0-862f-cbc9914f1daf" containerID="f7f9a444966779560bd7e61f9223e1bf4150cfec88bce66e73a12f404cb378fb" exitCode=0 Nov 03 22:37:39.242625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:39.242224 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" event={"ID":"0e3852f9-c306-4bb0-862f-cbc9914f1daf","Type":"ContainerDied","Data":"f7f9a444966779560bd7e61f9223e1bf4150cfec88bce66e73a12f404cb378fb"} Nov 03 22:37:40.422371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.422346 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:40.564593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.564513 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-xks-ca\") pod \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " Nov 03 22:37:40.564801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.564759 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-openshift-ca\") pod \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " Nov 03 22:37:40.564911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.564829 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3852f9-c306-4bb0-862f-cbc9914f1daf-kube-api-access-7zvgc\" (UniqueName: \"kubernetes.io/projected/0e3852f9-c306-4bb0-862f-cbc9914f1daf-kube-api-access-7zvgc\") pod \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\" (UID: \"0e3852f9-c306-4bb0-862f-cbc9914f1daf\") " Nov 03 22:37:40.565169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.565142 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-openshift-ca" pod "0e3852f9-c306-4bb0-862f-cbc9914f1daf" (UID: "0e3852f9-c306-4bb0-862f-cbc9914f1daf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:40.565577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.565557 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:40.567654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.567626 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-xks-ca" pod "0e3852f9-c306-4bb0-862f-cbc9914f1daf" (UID: "0e3852f9-c306-4bb0-862f-cbc9914f1daf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:40.567811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.567787 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3852f9-c306-4bb0-862f-cbc9914f1daf-kube-api-access-7zvgc" pod "0e3852f9-c306-4bb0-862f-cbc9914f1daf" (UID: "0e3852f9-c306-4bb0-862f-cbc9914f1daf"). InnerVolumeSpecName "kube-api-access-7zvgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:40.667483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.667451 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e3852f9-c306-4bb0-862f-cbc9914f1daf-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:40.667483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:40.667479 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvgc\" (UniqueName: \"kubernetes.io/projected/0e3852f9-c306-4bb0-862f-cbc9914f1daf-kube-api-access-7zvgc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:41.260151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:41.260121 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" Nov 03 22:37:41.260367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:41.260147 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv" event={"ID":"0e3852f9-c306-4bb0-862f-cbc9914f1daf","Type":"ContainerDied","Data":"ab36b44898eb3ab819bde0bbcbff0b038e35bda9925a771cc44cda614562ff25"} Nov 03 22:37:41.260367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:41.260201 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab36b44898eb3ab819bde0bbcbff0b038e35bda9925a771cc44cda614562ff25" Nov 03 22:37:43.901753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.901710 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq"] Nov 03 22:37:43.902690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.902668 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0e3852f9-c306-4bb0-862f-cbc9914f1daf" containerName="revoke-keys" Nov 03 22:37:43.902690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.902687 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3852f9-c306-4bb0-862f-cbc9914f1daf" containerName="revoke-keys" Nov 03 22:37:43.902875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.902840 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3852f9-c306-4bb0-862f-cbc9914f1daf" containerName="revoke-keys" Nov 03 22:37:43.907172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.907151 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:43.926214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.926182 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq"] Nov 03 22:37:43.929206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.929185 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:37:43.929372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.929209 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:37:43.929372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.929224 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:37:43.930781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:43.930758 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:37:44.021311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.021268 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.021664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.021630 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.021778 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.021684 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mk52\" (UniqueName: \"kubernetes.io/projected/38099760-6501-4a66-9a2d-8a1796e0eaee-kube-api-access-8mk52\") pod \"maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.123429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.123387 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.123625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.123468 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mk52\" (UniqueName: \"kubernetes.io/projected/38099760-6501-4a66-9a2d-8a1796e0eaee-kube-api-access-8mk52\") pod \"maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.123625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.123570 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.124085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.124067 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.125979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.125945 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.144802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.144773 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mk52\" (UniqueName: \"kubernetes.io/projected/38099760-6501-4a66-9a2d-8a1796e0eaee-kube-api-access-8mk52\") pod \"maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.225135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.225059 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:44.544615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:44.544585 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq"] Nov 03 22:37:44.544735 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:37:44.544653 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38099760_6501_4a66_9a2d_8a1796e0eaee.slice/crio-47b70e4fbcd586a80e836cb2ef728f118075d1c8987b1a026a96b9b601ba4e21 WatchSource:0}: Error finding container 47b70e4fbcd586a80e836cb2ef728f118075d1c8987b1a026a96b9b601ba4e21: Status 404 returned error can't find the container with id 47b70e4fbcd586a80e836cb2ef728f118075d1c8987b1a026a96b9b601ba4e21 Nov 03 22:37:45.308352 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:45.308309 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" event={"ID":"38099760-6501-4a66-9a2d-8a1796e0eaee","Type":"ContainerStarted","Data":"fb4b75e7dff2f988c2b296004892297b414e9966eed44ed5273bab05271ad68e"} Nov 03 22:37:45.308836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:45.308364 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" event={"ID":"38099760-6501-4a66-9a2d-8a1796e0eaee","Type":"ContainerStarted","Data":"47b70e4fbcd586a80e836cb2ef728f118075d1c8987b1a026a96b9b601ba4e21"} Nov 03 22:37:45.350625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:45.350569 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" podStartSLOduration=2.350555189 podStartE2EDuration="2.350555189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:37:45.346761681 +0000 UTC m=+1491.750702656" watchObservedRunningTime="2026-11-03 22:37:45.350555189 +0000 UTC m=+1491.754496163" Nov 03 22:37:46.318842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:46.318810 4903 generic.go:356] "Generic (PLEG): container finished" podID="38099760-6501-4a66-9a2d-8a1796e0eaee" containerID="fb4b75e7dff2f988c2b296004892297b414e9966eed44ed5273bab05271ad68e" exitCode=0 Nov 03 22:37:46.319275 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:46.318889 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" event={"ID":"38099760-6501-4a66-9a2d-8a1796e0eaee","Type":"ContainerDied","Data":"fb4b75e7dff2f988c2b296004892297b414e9966eed44ed5273bab05271ad68e"} Nov 03 22:37:47.506205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.506173 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:47.606923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.606825 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-openshift-ca\") pod \"38099760-6501-4a66-9a2d-8a1796e0eaee\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " Nov 03 22:37:47.607404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.607378 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38099760-6501-4a66-9a2d-8a1796e0eaee-kube-api-access-8mk52\" (UniqueName: \"kubernetes.io/projected/38099760-6501-4a66-9a2d-8a1796e0eaee-kube-api-access-8mk52\") pod \"38099760-6501-4a66-9a2d-8a1796e0eaee\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " Nov 03 22:37:47.607667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.607646 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-xks-ca\") pod \"38099760-6501-4a66-9a2d-8a1796e0eaee\" (UID: \"38099760-6501-4a66-9a2d-8a1796e0eaee\") " Nov 03 22:37:47.634342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.607445 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-openshift-ca" pod "38099760-6501-4a66-9a2d-8a1796e0eaee" (UID: "38099760-6501-4a66-9a2d-8a1796e0eaee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:47.657371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.654530 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38099760-6501-4a66-9a2d-8a1796e0eaee-kube-api-access-8mk52" pod "38099760-6501-4a66-9a2d-8a1796e0eaee" (UID: "38099760-6501-4a66-9a2d-8a1796e0eaee"). InnerVolumeSpecName "kube-api-access-8mk52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:47.779194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.760625 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:47.779194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.760660 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mk52\" (UniqueName: \"kubernetes.io/projected/38099760-6501-4a66-9a2d-8a1796e0eaee-kube-api-access-8mk52\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:47.822535 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.820227 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-xks-ca" pod "38099760-6501-4a66-9a2d-8a1796e0eaee" (UID: "38099760-6501-4a66-9a2d-8a1796e0eaee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:37:47.862533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:47.862446 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38099760-6501-4a66-9a2d-8a1796e0eaee-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:48.059445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:48.059405 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx"] Nov 03 22:37:48.342389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:48.342352 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" event={"ID":"38099760-6501-4a66-9a2d-8a1796e0eaee","Type":"ContainerDied","Data":"47b70e4fbcd586a80e836cb2ef728f118075d1c8987b1a026a96b9b601ba4e21"} Nov 03 22:37:48.342576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:48.342416 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47b70e4fbcd586a80e836cb2ef728f118075d1c8987b1a026a96b9b601ba4e21" Nov 03 22:37:48.342576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:48.342425 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq" Nov 03 22:37:48.387671 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:37:48.387635 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:37:53.059854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.059809 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" podUID="91626678-bacb-47cd-bc41-69ebdea13346" containerName="payload-pre-processing" containerID="cri-o://61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9" gracePeriod=25 Nov 03 22:37:53.406732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.406697 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:37:53.417921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.417887 4903 generic.go:356] "Generic (PLEG): container finished" podID="91626678-bacb-47cd-bc41-69ebdea13346" containerID="61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9" exitCode=0 Nov 03 22:37:53.418104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.417962 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" Nov 03 22:37:53.418104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.418019 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" event={"ID":"91626678-bacb-47cd-bc41-69ebdea13346","Type":"ContainerDied","Data":"61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9"} Nov 03 22:37:53.418104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.418097 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx" event={"ID":"91626678-bacb-47cd-bc41-69ebdea13346","Type":"ContainerDied","Data":"2a72c0c3539e61695d29982b3365429521a339e99beb5d6b1a1d639b696a4193"} Nov 03 22:37:53.418278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.418127 4903 scope.go:122] "RemoveContainer" containerID="61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9" Nov 03 22:37:53.434943 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.434919 4903 scope.go:122] "RemoveContainer" containerID="61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9" Nov 03 22:37:53.435287 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:37:53.435264 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9\": container with ID starting with 61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9 not found: ID does not exist" containerID="61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9" Nov 03 22:37:53.435444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.435324 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9"} err="failed to get container status \"61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9\": rpc error: code = NotFound desc = could not find container \"61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9\": container with ID starting with 61b3c59c3c3d3fd29cc0d1e66d53d2b9360dfe9bb75643e0eb12aee2fb9008f9 not found: ID does not exist" Nov 03 22:37:53.579196 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.579165 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91626678-bacb-47cd-bc41-69ebdea13346-kube-api-access-8bljj\" (UniqueName: \"kubernetes.io/projected/91626678-bacb-47cd-bc41-69ebdea13346-kube-api-access-8bljj\") pod \"91626678-bacb-47cd-bc41-69ebdea13346\" (UID: \"91626678-bacb-47cd-bc41-69ebdea13346\") " Nov 03 22:37:53.579413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.579255 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91626678-bacb-47cd-bc41-69ebdea13346-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91626678-bacb-47cd-bc41-69ebdea13346-plugins-config-volume\") pod \"91626678-bacb-47cd-bc41-69ebdea13346\" (UID: \"91626678-bacb-47cd-bc41-69ebdea13346\") " Nov 03 22:37:53.579642 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.579612 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91626678-bacb-47cd-bc41-69ebdea13346-plugins-config-volume" pod "91626678-bacb-47cd-bc41-69ebdea13346" (UID: "91626678-bacb-47cd-bc41-69ebdea13346"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:37:53.579955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.579938 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91626678-bacb-47cd-bc41-69ebdea13346-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:53.581493 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.581467 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91626678-bacb-47cd-bc41-69ebdea13346-kube-api-access-8bljj" pod "91626678-bacb-47cd-bc41-69ebdea13346" (UID: "91626678-bacb-47cd-bc41-69ebdea13346"). InnerVolumeSpecName "kube-api-access-8bljj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:37:53.681470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.681434 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bljj\" (UniqueName: \"kubernetes.io/projected/91626678-bacb-47cd-bc41-69ebdea13346-kube-api-access-8bljj\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:37:53.768727 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.768700 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx"] Nov 03 22:37:53.779357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:53.779336 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bbfc2e-78f95f4d55-5tprx"] Nov 03 22:37:54.057160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:54.057127 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:37:54.057501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:54.057481 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:37:54.070359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:54.070329 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:37:54.070889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:54.070860 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:37:54.624243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:54.623969 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6"] Nov 03 22:37:54.652692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:54.652661 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq"] Nov 03 22:37:55.741149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:55.740796 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91626678-bacb-47cd-bc41-69ebdea13346" path="/var/lib/kubelet/pods/91626678-bacb-47cd-bc41-69ebdea13346/volumes" Nov 03 22:37:57.189145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:57.180765 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:37:59.627356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:59.626349 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" podUID="0d4d41e8-3176-4768-84ff-2aae8a277e5f" containerName="payload-processing" containerID="cri-o://f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283" gracePeriod=25 Nov 03 22:37:59.653257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:37:59.653219 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" podUID="23fc63a8-8b2a-4eca-9d24-74e727a8e3c9" containerName="payload-pre-processing" containerID="cri-o://886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93" gracePeriod=25 Nov 03 22:38:00.024364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.024337 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:38:00.063029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.063003 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:38:00.131843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.131810 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-plugins-config-volume\") pod \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\" (UID: \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\") " Nov 03 22:38:00.132013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.131866 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4d41e8-3176-4768-84ff-2aae8a277e5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d41e8-3176-4768-84ff-2aae8a277e5f-plugins-config-volume\") pod \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\" (UID: \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\") " Nov 03 22:38:00.132079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.132041 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-kube-api-access-nm6pf\" (UniqueName: \"kubernetes.io/projected/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-kube-api-access-nm6pf\") pod \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\" (UID: \"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9\") " Nov 03 22:38:00.132173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.132156 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4d41e8-3176-4768-84ff-2aae8a277e5f-kube-api-access-57z4l\" (UniqueName: \"kubernetes.io/projected/0d4d41e8-3176-4768-84ff-2aae8a277e5f-kube-api-access-57z4l\") pod \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\" (UID: \"0d4d41e8-3176-4768-84ff-2aae8a277e5f\") " Nov 03 22:38:00.132230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.132217 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4d41e8-3176-4768-84ff-2aae8a277e5f-plugins-config-volume" pod "0d4d41e8-3176-4768-84ff-2aae8a277e5f" (UID: "0d4d41e8-3176-4768-84ff-2aae8a277e5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:00.132326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.132268 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-plugins-config-volume" pod "23fc63a8-8b2a-4eca-9d24-74e727a8e3c9" (UID: "23fc63a8-8b2a-4eca-9d24-74e727a8e3c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:00.133404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.133377 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:00.133404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.133406 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4d41e8-3176-4768-84ff-2aae8a277e5f-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:00.134524 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.134494 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-kube-api-access-nm6pf" pod "23fc63a8-8b2a-4eca-9d24-74e727a8e3c9" (UID: "23fc63a8-8b2a-4eca-9d24-74e727a8e3c9"). InnerVolumeSpecName "kube-api-access-nm6pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:00.135173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.135152 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4d41e8-3176-4768-84ff-2aae8a277e5f-kube-api-access-57z4l" pod "0d4d41e8-3176-4768-84ff-2aae8a277e5f" (UID: "0d4d41e8-3176-4768-84ff-2aae8a277e5f"). InnerVolumeSpecName "kube-api-access-57z4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:00.235347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.235315 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm6pf\" (UniqueName: \"kubernetes.io/projected/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9-kube-api-access-nm6pf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:00.235347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.235341 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57z4l\" (UniqueName: \"kubernetes.io/projected/0d4d41e8-3176-4768-84ff-2aae8a277e5f-kube-api-access-57z4l\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:00.559375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.554281 4903 generic.go:356] "Generic (PLEG): container finished" podID="23fc63a8-8b2a-4eca-9d24-74e727a8e3c9" containerID="886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93" exitCode=0 Nov 03 22:38:00.559375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.554416 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" event={"ID":"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9","Type":"ContainerDied","Data":"886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93"} Nov 03 22:38:00.559375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.554473 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" event={"ID":"23fc63a8-8b2a-4eca-9d24-74e727a8e3c9","Type":"ContainerDied","Data":"bccbd2fec40e5d657e766414979e99fa877fbbeac4ad505229525bdf23e3a7df"} Nov 03 22:38:00.559375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.554507 4903 scope.go:122] "RemoveContainer" containerID="886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93" Nov 03 22:38:00.559375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.554661 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq" Nov 03 22:38:00.566081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.566043 4903 generic.go:356] "Generic (PLEG): container finished" podID="0d4d41e8-3176-4768-84ff-2aae8a277e5f" containerID="f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283" exitCode=0 Nov 03 22:38:00.566229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.566182 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" Nov 03 22:38:00.566229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.566214 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" event={"ID":"0d4d41e8-3176-4768-84ff-2aae8a277e5f","Type":"ContainerDied","Data":"f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283"} Nov 03 22:38:00.566366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.566260 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6" event={"ID":"0d4d41e8-3176-4768-84ff-2aae8a277e5f","Type":"ContainerDied","Data":"dfb1c2ba9d5ed72e683d5a1b56891ddf64650b2f88b21c8a6cfa43a5066c491f"} Nov 03 22:38:00.580930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.580909 4903 scope.go:122] "RemoveContainer" containerID="886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93" Nov 03 22:38:00.581270 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:00.581247 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93\": container with ID starting with 886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93 not found: ID does not exist" containerID="886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93" Nov 03 22:38:00.581374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.581277 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93"} err="failed to get container status \"886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93\": rpc error: code = NotFound desc = could not find container \"886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93\": container with ID starting with 886adfcd5ed5a466ba14703e428daceafe07a20431e5da31fd74a7abf6f24e93 not found: ID does not exist" Nov 03 22:38:00.581374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.581353 4903 scope.go:122] "RemoveContainer" containerID="f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283" Nov 03 22:38:00.592046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.592020 4903 scope.go:122] "RemoveContainer" containerID="f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283" Nov 03 22:38:00.592411 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:00.592391 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283\": container with ID starting with f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283 not found: ID does not exist" containerID="f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283" Nov 03 22:38:00.592490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.592419 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283"} err="failed to get container status \"f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283\": rpc error: code = NotFound desc = could not find container \"f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283\": container with ID starting with f89d4de564c4bff1fe48647ef1e50de3498a07f30478b3fed4a005a0ae822283 not found: ID does not exist" Nov 03 22:38:00.622687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.622659 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq"] Nov 03 22:38:00.628783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.628756 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-32fdf2-6767d498c5-np8rq"] Nov 03 22:38:00.673185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.673155 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6"] Nov 03 22:38:00.678342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:00.678307 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-32fdf2-7469cf674d-4c2l6"] Nov 03 22:38:01.742238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:01.742206 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4d41e8-3176-4768-84ff-2aae8a277e5f" path="/var/lib/kubelet/pods/0d4d41e8-3176-4768-84ff-2aae8a277e5f/volumes" Nov 03 22:38:01.742869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:01.742850 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fc63a8-8b2a-4eca-9d24-74e727a8e3c9" path="/var/lib/kubelet/pods/23fc63a8-8b2a-4eca-9d24-74e727a8e3c9/volumes" Nov 03 22:38:02.031975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.031891 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps"] Nov 03 22:38:02.042356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.042326 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9865da23-stbps"] Nov 03 22:38:02.853341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.853307 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp"] Nov 03 22:38:02.854204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854187 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0d4d41e8-3176-4768-84ff-2aae8a277e5f" containerName="payload-processing" Nov 03 22:38:02.854258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854205 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4d41e8-3176-4768-84ff-2aae8a277e5f" containerName="payload-processing" Nov 03 22:38:02.854258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854229 4903 cpu_manager.go:414] "containerMap: removing container" podUID="38099760-6501-4a66-9a2d-8a1796e0eaee" containerName="revoke-keys" Nov 03 22:38:02.854258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854234 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38099760-6501-4a66-9a2d-8a1796e0eaee" containerName="revoke-keys" Nov 03 22:38:02.854258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854251 4903 cpu_manager.go:414] "containerMap: removing container" podUID="23fc63a8-8b2a-4eca-9d24-74e727a8e3c9" containerName="payload-pre-processing" Nov 03 22:38:02.854258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854256 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fc63a8-8b2a-4eca-9d24-74e727a8e3c9" containerName="payload-pre-processing" Nov 03 22:38:02.854453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854262 4903 cpu_manager.go:414] "containerMap: removing container" podUID="91626678-bacb-47cd-bc41-69ebdea13346" containerName="payload-pre-processing" Nov 03 22:38:02.854453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854268 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91626678-bacb-47cd-bc41-69ebdea13346" containerName="payload-pre-processing" Nov 03 22:38:02.854516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854491 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4d41e8-3176-4768-84ff-2aae8a277e5f" containerName="payload-processing" Nov 03 22:38:02.854516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854505 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="38099760-6501-4a66-9a2d-8a1796e0eaee" containerName="revoke-keys" Nov 03 22:38:02.854576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854511 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fc63a8-8b2a-4eca-9d24-74e727a8e3c9" containerName="payload-pre-processing" Nov 03 22:38:02.854576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.854522 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="91626678-bacb-47cd-bc41-69ebdea13346" containerName="payload-pre-processing" Nov 03 22:38:02.860142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.860117 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:02.877152 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.877121 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp"] Nov 03 22:38:02.880466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.880421 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:38:02.880466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.880454 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:02.884363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.884342 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:38:02.884491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:02.884395 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:38:03.014929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.014897 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.015104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.014944 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.015195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.015171 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c722l\" (UniqueName: \"kubernetes.io/projected/d89c212b-9dd9-426e-9bb1-4a0652035966-kube-api-access-c722l\") pod \"maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.116599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.116521 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c722l\" (UniqueName: \"kubernetes.io/projected/d89c212b-9dd9-426e-9bb1-4a0652035966-kube-api-access-c722l\") pod \"maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.116775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.116754 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.116822 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.116797 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.117466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.117445 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.119242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.119219 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.138438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.138413 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c722l\" (UniqueName: \"kubernetes.io/projected/d89c212b-9dd9-426e-9bb1-4a0652035966-kube-api-access-c722l\") pod \"maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.178145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.178115 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:03.497204 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:38:03.497174 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89c212b_9dd9_426e_9bb1_4a0652035966.slice/crio-6d6387412ece91b5d4c01736c1c54959e2ed34e5cd67866e7c7ab06d247c40c7 WatchSource:0}: Error finding container 6d6387412ece91b5d4c01736c1c54959e2ed34e5cd67866e7c7ab06d247c40c7: Status 404 returned error can't find the container with id 6d6387412ece91b5d4c01736c1c54959e2ed34e5cd67866e7c7ab06d247c40c7 Nov 03 22:38:03.498968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.498872 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp"] Nov 03 22:38:03.597621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.597584 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" event={"ID":"d89c212b-9dd9-426e-9bb1-4a0652035966","Type":"ContainerStarted","Data":"6d6387412ece91b5d4c01736c1c54959e2ed34e5cd67866e7c7ab06d247c40c7"} Nov 03 22:38:03.737933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:03.737888 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253fa807-c223-4e87-8275-dc76cac0df01" path="/var/lib/kubelet/pods/253fa807-c223-4e87-8275-dc76cac0df01/volumes" Nov 03 22:38:04.520374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.520339 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-043c70"] Nov 03 22:38:04.525428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.525409 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:04.542961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.542932 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-043c70"] Nov 03 22:38:04.545165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.545127 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-khmwt\"" Nov 03 22:38:04.607755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.607662 4903 generic.go:356] "Generic (PLEG): container finished" podID="d89c212b-9dd9-426e-9bb1-4a0652035966" containerID="51c0ea122a59378cd8a044db4daec8cb335ba11483e62036c47323118ee2b612" exitCode=0 Nov 03 22:38:04.607755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.607733 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" event={"ID":"d89c212b-9dd9-426e-9bb1-4a0652035966","Type":"ContainerDied","Data":"51c0ea122a59378cd8a044db4daec8cb335ba11483e62036c47323118ee2b612"} Nov 03 22:38:04.644779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.644746 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltd45\" (UniqueName: \"kubernetes.io/projected/5a9a283e-c248-44f1-bbd3-24bead0a967c-kube-api-access-ltd45\") pod \"test-curl-7257-043c70\" (UID: \"5a9a283e-c248-44f1-bbd3-24bead0a967c\") " pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:04.746102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.746068 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltd45\" (UniqueName: \"kubernetes.io/projected/5a9a283e-c248-44f1-bbd3-24bead0a967c-kube-api-access-ltd45\") pod \"test-curl-7257-043c70\" (UID: \"5a9a283e-c248-44f1-bbd3-24bead0a967c\") " pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:04.767888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.767858 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltd45\" (UniqueName: \"kubernetes.io/projected/5a9a283e-c248-44f1-bbd3-24bead0a967c-kube-api-access-ltd45\") pod \"test-curl-7257-043c70\" (UID: \"5a9a283e-c248-44f1-bbd3-24bead0a967c\") " pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:04.844473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:04.844444 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:05.152689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.152662 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-043c70"] Nov 03 22:38:05.153408 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:38:05.153389 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a9a283e_c248_44f1_bbd3_24bead0a967c.slice/crio-7c9129a1148656af9bff4be5a8106091d58b1699cc9494e2ee27c9780e71e145 WatchSource:0}: Error finding container 7c9129a1148656af9bff4be5a8106091d58b1699cc9494e2ee27c9780e71e145: Status 404 returned error can't find the container with id 7c9129a1148656af9bff4be5a8106091d58b1699cc9494e2ee27c9780e71e145 Nov 03 22:38:05.619835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.619786 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-043c70" event={"ID":"5a9a283e-c248-44f1-bbd3-24bead0a967c","Type":"ContainerStarted","Data":"a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5"} Nov 03 22:38:05.620340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.619850 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-043c70" event={"ID":"5a9a283e-c248-44f1-bbd3-24bead0a967c","Type":"ContainerStarted","Data":"7c9129a1148656af9bff4be5a8106091d58b1699cc9494e2ee27c9780e71e145"} Nov 03 22:38:05.660475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.660423 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-043c70" podStartSLOduration=1.6604073289999999 podStartE2EDuration="1.660407329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:05.654215479 +0000 UTC m=+1512.058156457" watchObservedRunningTime="2026-11-03 22:38:05.660407329 +0000 UTC m=+1512.064348304" Nov 03 22:38:05.808438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.808411 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:05.866207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.866166 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89c212b-9dd9-426e-9bb1-4a0652035966-kube-api-access-c722l\" (UniqueName: \"kubernetes.io/projected/d89c212b-9dd9-426e-9bb1-4a0652035966-kube-api-access-c722l\") pod \"d89c212b-9dd9-426e-9bb1-4a0652035966\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " Nov 03 22:38:05.866442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.866426 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-openshift-ca\") pod \"d89c212b-9dd9-426e-9bb1-4a0652035966\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " Nov 03 22:38:05.866521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.866483 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-xks-ca\") pod \"d89c212b-9dd9-426e-9bb1-4a0652035966\" (UID: \"d89c212b-9dd9-426e-9bb1-4a0652035966\") " Nov 03 22:38:05.866875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.866850 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-openshift-ca" pod "d89c212b-9dd9-426e-9bb1-4a0652035966" (UID: "d89c212b-9dd9-426e-9bb1-4a0652035966"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:05.867109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.867088 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:05.869002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.868967 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89c212b-9dd9-426e-9bb1-4a0652035966-kube-api-access-c722l" pod "d89c212b-9dd9-426e-9bb1-4a0652035966" (UID: "d89c212b-9dd9-426e-9bb1-4a0652035966"). InnerVolumeSpecName "kube-api-access-c722l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:05.869129 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.869049 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-xks-ca" pod "d89c212b-9dd9-426e-9bb1-4a0652035966" (UID: "d89c212b-9dd9-426e-9bb1-4a0652035966"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:05.969016 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.968983 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89c212b-9dd9-426e-9bb1-4a0652035966-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:05.969016 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:05.969013 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c722l\" (UniqueName: \"kubernetes.io/projected/d89c212b-9dd9-426e-9bb1-4a0652035966-kube-api-access-c722l\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:06.559854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.559818 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-043c70"] Nov 03 22:38:06.564933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.564902 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-043c70"] Nov 03 22:38:06.631214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.631184 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" Nov 03 22:38:06.631730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.631208 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp" event={"ID":"d89c212b-9dd9-426e-9bb1-4a0652035966","Type":"ContainerDied","Data":"6d6387412ece91b5d4c01736c1c54959e2ed34e5cd67866e7c7ab06d247c40c7"} Nov 03 22:38:06.631730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.631266 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d6387412ece91b5d4c01736c1c54959e2ed34e5cd67866e7c7ab06d247c40c7" Nov 03 22:38:06.904061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.904033 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l"] Nov 03 22:38:06.904909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.904892 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d89c212b-9dd9-426e-9bb1-4a0652035966" containerName="revoke-keys" Nov 03 22:38:06.904958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.904910 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89c212b-9dd9-426e-9bb1-4a0652035966" containerName="revoke-keys" Nov 03 22:38:06.904958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.904926 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5a9a283e-c248-44f1-bbd3-24bead0a967c" containerName="test-curl-7257-043c70" Nov 03 22:38:06.904958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.904932 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9a283e-c248-44f1-bbd3-24bead0a967c" containerName="test-curl-7257-043c70" Nov 03 22:38:06.905106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.905097 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89c212b-9dd9-426e-9bb1-4a0652035966" containerName="revoke-keys" Nov 03 22:38:06.905141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.905110 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9a283e-c248-44f1-bbd3-24bead0a967c" containerName="test-curl-7257-043c70" Nov 03 22:38:06.909055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.909036 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:06.923379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.922584 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:38:06.923529 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.923395 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:38:06.927488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.927462 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l"] Nov 03 22:38:06.971600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.971562 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:06.971843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.971823 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:38:06.986249 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.986213 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwd6\" (UniqueName: \"kubernetes.io/projected/31ef30f9-6a5b-435a-b572-9cda64ff9d14-kube-api-access-qgwd6\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:06.986645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.986619 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:06.986876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:06.986849 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:07.088272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.088231 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:07.088528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.088410 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:07.088528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.088478 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwd6\" (UniqueName: \"kubernetes.io/projected/31ef30f9-6a5b-435a-b572-9cda64ff9d14-kube-api-access-qgwd6\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:07.089014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.088994 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:07.090614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.090592 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:07.109891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.109856 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwd6\" (UniqueName: \"kubernetes.io/projected/31ef30f9-6a5b-435a-b572-9cda64ff9d14-kube-api-access-qgwd6\") pod \"maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:07.227552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.227480 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:07.547633 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:38:07.547598 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ef30f9_6a5b_435a_b572_9cda64ff9d14.slice/crio-a995c513191996ef8ce8abd2ad85bf85c4a6fd9fae257f2bda70f5d98adafeb0 WatchSource:0}: Error finding container a995c513191996ef8ce8abd2ad85bf85c4a6fd9fae257f2bda70f5d98adafeb0: Status 404 returned error can't find the container with id a995c513191996ef8ce8abd2ad85bf85c4a6fd9fae257f2bda70f5d98adafeb0 Nov 03 22:38:07.547923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.547904 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l"] Nov 03 22:38:07.643655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.643614 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" event={"ID":"31ef30f9-6a5b-435a-b572-9cda64ff9d14","Type":"ContainerStarted","Data":"a995c513191996ef8ce8abd2ad85bf85c4a6fd9fae257f2bda70f5d98adafeb0"} Nov 03 22:38:07.644127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:07.643710 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-043c70" podUID="5a9a283e-c248-44f1-bbd3-24bead0a967c" containerName="test-curl-7257-043c70" containerID="cri-o://a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5" gracePeriod=2 Nov 03 22:38:07.655770 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:07.655740 4903 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-043c70\" is forbidden: User \"system:node:ip-10-0-133-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-193.ec2.internal' and this object" podUID="5a9a283e-c248-44f1-bbd3-24bead0a967c" pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:08.655796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:08.655763 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" event={"ID":"31ef30f9-6a5b-435a-b572-9cda64ff9d14","Type":"ContainerStarted","Data":"c238b9dd2e8c7c62029d99b833895eed2591f2869806e8b5b074c20c2de4c842"} Nov 03 22:38:09.667765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.667734 4903 generic.go:356] "Generic (PLEG): container finished" podID="31ef30f9-6a5b-435a-b572-9cda64ff9d14" containerID="c238b9dd2e8c7c62029d99b833895eed2591f2869806e8b5b074c20c2de4c842" exitCode=0 Nov 03 22:38:09.668133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.667826 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" event={"ID":"31ef30f9-6a5b-435a-b572-9cda64ff9d14","Type":"ContainerDied","Data":"c238b9dd2e8c7c62029d99b833895eed2591f2869806e8b5b074c20c2de4c842"} Nov 03 22:38:09.867145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.867115 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:09.932580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.932550 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:09.964919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.964884 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-xks-ca\") pod \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " Nov 03 22:38:09.965121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.965097 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-openshift-ca\") pod \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " Nov 03 22:38:09.965185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.965128 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ef30f9-6a5b-435a-b572-9cda64ff9d14-kube-api-access-qgwd6\" (UniqueName: \"kubernetes.io/projected/31ef30f9-6a5b-435a-b572-9cda64ff9d14-kube-api-access-qgwd6\") pod \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\" (UID: \"31ef30f9-6a5b-435a-b572-9cda64ff9d14\") " Nov 03 22:38:09.965604 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.965571 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-openshift-ca" pod "31ef30f9-6a5b-435a-b572-9cda64ff9d14" (UID: "31ef30f9-6a5b-435a-b572-9cda64ff9d14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:09.965799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.965781 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:09.967258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.967235 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-xks-ca" pod "31ef30f9-6a5b-435a-b572-9cda64ff9d14" (UID: "31ef30f9-6a5b-435a-b572-9cda64ff9d14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:09.967359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:09.967335 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ef30f9-6a5b-435a-b572-9cda64ff9d14-kube-api-access-qgwd6" pod "31ef30f9-6a5b-435a-b572-9cda64ff9d14" (UID: "31ef30f9-6a5b-435a-b572-9cda64ff9d14"). InnerVolumeSpecName "kube-api-access-qgwd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:10.067338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.067302 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9a283e-c248-44f1-bbd3-24bead0a967c-kube-api-access-ltd45\" (UniqueName: \"kubernetes.io/projected/5a9a283e-c248-44f1-bbd3-24bead0a967c-kube-api-access-ltd45\") pod \"5a9a283e-c248-44f1-bbd3-24bead0a967c\" (UID: \"5a9a283e-c248-44f1-bbd3-24bead0a967c\") " Nov 03 22:38:10.068107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.068088 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgwd6\" (UniqueName: \"kubernetes.io/projected/31ef30f9-6a5b-435a-b572-9cda64ff9d14-kube-api-access-qgwd6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:10.068149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.068111 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ef30f9-6a5b-435a-b572-9cda64ff9d14-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:10.069446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.069423 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9a283e-c248-44f1-bbd3-24bead0a967c-kube-api-access-ltd45" pod "5a9a283e-c248-44f1-bbd3-24bead0a967c" (UID: "5a9a283e-c248-44f1-bbd3-24bead0a967c"). InnerVolumeSpecName "kube-api-access-ltd45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:10.169974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.169900 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltd45\" (UniqueName: \"kubernetes.io/projected/5a9a283e-c248-44f1-bbd3-24bead0a967c-kube-api-access-ltd45\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:10.710476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.710376 4903 generic.go:356] "Generic (PLEG): container finished" podID="5a9a283e-c248-44f1-bbd3-24bead0a967c" containerID="a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5" exitCode=137 Nov 03 22:38:10.710476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.710429 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:10.711356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.711323 4903 scope.go:122] "RemoveContainer" containerID="a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5" Nov 03 22:38:10.713343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.713326 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" Nov 03 22:38:10.713452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.713354 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l" event={"ID":"31ef30f9-6a5b-435a-b572-9cda64ff9d14","Type":"ContainerDied","Data":"a995c513191996ef8ce8abd2ad85bf85c4a6fd9fae257f2bda70f5d98adafeb0"} Nov 03 22:38:10.713452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.713408 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a995c513191996ef8ce8abd2ad85bf85c4a6fd9fae257f2bda70f5d98adafeb0" Nov 03 22:38:10.726381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.726360 4903 scope.go:122] "RemoveContainer" containerID="a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5" Nov 03 22:38:10.726743 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:10.726714 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5\": container with ID starting with a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5 not found: ID does not exist" containerID="a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5" Nov 03 22:38:10.726822 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:10.726754 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5"} err="failed to get container status \"a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5\": rpc error: code = NotFound desc = could not find container \"a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5\": container with ID starting with a1d8dcfdf37927103a2a38573ddeaa49f342e55083dc3960487b791d555778b5 not found: ID does not exist" Nov 03 22:38:10.735211 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:10.735182 4903 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-043c70\" is forbidden: User \"system:node:ip-10-0-133-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-193.ec2.internal' and this object" podUID="5a9a283e-c248-44f1-bbd3-24bead0a967c" pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:10.759754 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:10.759719 4903 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-043c70\" is forbidden: User \"system:node:ip-10-0-133-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-193.ec2.internal' and this object" podUID="5a9a283e-c248-44f1-bbd3-24bead0a967c" pod="opendatahub/test-curl-7257-043c70" Nov 03 22:38:11.739219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:11.739180 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9a283e-c248-44f1-bbd3-24bead0a967c" path="/var/lib/kubelet/pods/5a9a283e-c248-44f1-bbd3-24bead0a967c/volumes" Nov 03 22:38:17.525351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:17.525319 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct"] Nov 03 22:38:17.555617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:17.555586 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz"] Nov 03 22:38:18.393793 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:18.393751 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:38:20.507915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.507877 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2"] Nov 03 22:38:20.508947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.508924 4903 cpu_manager.go:414] "containerMap: removing container" podUID="31ef30f9-6a5b-435a-b572-9cda64ff9d14" containerName="revoke-keys" Nov 03 22:38:20.509062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.508947 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ef30f9-6a5b-435a-b572-9cda64ff9d14" containerName="revoke-keys" Nov 03 22:38:20.509223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.509204 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ef30f9-6a5b-435a-b572-9cda64ff9d14" containerName="revoke-keys" Nov 03 22:38:20.513637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.513580 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.531115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.531085 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b1ec3a-openshift-default-dockercfg-p4w6w\"" Nov 03 22:38:20.543370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.543345 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2"] Nov 03 22:38:20.572412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.572377 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-certs\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.572578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.572433 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-podinfo\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.572578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.572493 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-credential-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.572578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.572552 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-data\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.572690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.572647 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-token\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.572690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.572681 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-envoy\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.572796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.572781 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtlp\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-kube-api-access-pxtlp\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.572902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.572879 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.573047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.572918 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istiod-ca-cert\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.686706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.686646 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-certs\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.687034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.687008 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-podinfo\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.720103 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.720069 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-credential-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.720441 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.720414 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-data\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.720716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.720699 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-token\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.720857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.720839 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-envoy\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.721079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.721062 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtlp\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-kube-api-access-pxtlp\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.722502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.722470 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.722669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.722652 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istiod-ca-cert\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.722936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.721732 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-credential-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.723048 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.687634 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-certs\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.723316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.723264 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-socket\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.723748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.721508 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-data\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.758959 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.738967 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istiod-ca-cert\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.768737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.764871 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-podinfo\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.782317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.773864 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtlp\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-kube-api-access-pxtlp\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.782317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.777726 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-envoy\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.783447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.782950 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-token\") pod \"e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:20.856316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:20.852634 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:21.236365 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:38:21.236332 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6694caf3_1707_4ae5_a4b8_d9bc3e16ce8e.slice/crio-c75a8e78a7477215547f14c913b2e0d4e6b0048b232b62455caf4c622705b7c3 WatchSource:0}: Error finding container c75a8e78a7477215547f14c913b2e0d4e6b0048b232b62455caf4c622705b7c3: Status 404 returned error can't find the container with id c75a8e78a7477215547f14c913b2e0d4e6b0048b232b62455caf4c622705b7c3 Nov 03 22:38:21.238197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:21.238055 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2"] Nov 03 22:38:21.239977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:21.239935 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:38:21.240105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:21.240022 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:38:21.240105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:21.240065 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:38:21.914150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:21.914113 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" event={"ID":"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e","Type":"ContainerStarted","Data":"349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841"} Nov 03 22:38:21.914569 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:21.914168 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" event={"ID":"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e","Type":"ContainerStarted","Data":"c75a8e78a7477215547f14c913b2e0d4e6b0048b232b62455caf4c622705b7c3"} Nov 03 22:38:22.525908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.525844 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" podUID="8d5d8452-9a8e-4422-816c-e5ab42caa1bb" containerName="payload-processing" containerID="cri-o://c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c" gracePeriod=25 Nov 03 22:38:22.556675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.556644 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" podUID="152ffdb2-bb93-4293-a4ae-0ec17515b0d6" containerName="payload-pre-processing" containerID="cri-o://df38ef3bf3292b61be3d59590aa04108d2801ff674c500f40f63a271581f011b" gracePeriod=25 Nov 03 22:38:22.854130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.854083 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:22.866186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.866154 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:22.878027 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.877990 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:38:22.923206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.923145 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" podStartSLOduration=2.923127409 podStartE2EDuration="2.923127409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:21.976363155 +0000 UTC m=+1528.380304130" watchObservedRunningTime="2026-11-03 22:38:22.923127409 +0000 UTC m=+1529.327068385" Nov 03 22:38:22.932061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.932028 4903 generic.go:356] "Generic (PLEG): container finished" podID="8d5d8452-9a8e-4422-816c-e5ab42caa1bb" containerID="c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c" exitCode=0 Nov 03 22:38:22.932213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.932176 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" event={"ID":"8d5d8452-9a8e-4422-816c-e5ab42caa1bb","Type":"ContainerDied","Data":"c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c"} Nov 03 22:38:22.932285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.932226 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" event={"ID":"8d5d8452-9a8e-4422-816c-e5ab42caa1bb","Type":"ContainerDied","Data":"c625183428b44cd71c33d2030161ca72093e4e9869eb6670f7f01062a1f2de77"} Nov 03 22:38:22.932285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.932256 4903 scope.go:122] "RemoveContainer" containerID="c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c" Nov 03 22:38:22.932463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.932411 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct" Nov 03 22:38:22.934081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.934053 4903 generic.go:356] "Generic (PLEG): container finished" podID="152ffdb2-bb93-4293-a4ae-0ec17515b0d6" containerID="df38ef3bf3292b61be3d59590aa04108d2801ff674c500f40f63a271581f011b" exitCode=0 Nov 03 22:38:22.934200 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.934094 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" event={"ID":"152ffdb2-bb93-4293-a4ae-0ec17515b0d6","Type":"ContainerDied","Data":"df38ef3bf3292b61be3d59590aa04108d2801ff674c500f40f63a271581f011b"} Nov 03 22:38:22.934270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.934194 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" event={"ID":"152ffdb2-bb93-4293-a4ae-0ec17515b0d6","Type":"ContainerDied","Data":"75a52bfe80206af2329ab198f3d6e1bc4c0d0b521730de4316a273f6c13dd02d"} Nov 03 22:38:22.934270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.934224 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75a52bfe80206af2329ab198f3d6e1bc4c0d0b521730de4316a273f6c13dd02d" Nov 03 22:38:22.934767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.934749 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:22.936098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.936081 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:38:22.945952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.945930 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:38:22.949123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.949105 4903 scope.go:122] "RemoveContainer" containerID="c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c" Nov 03 22:38:22.949456 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:22.949436 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c\": container with ID starting with c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c not found: ID does not exist" containerID="c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c" Nov 03 22:38:22.949508 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.949467 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c"} err="failed to get container status \"c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c\": rpc error: code = NotFound desc = could not find container \"c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c\": container with ID starting with c32e2976c2198130450aae7c0f57634eae9df195f60aa9500f6515c778d9b81c not found: ID does not exist" Nov 03 22:38:22.966828 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.966802 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-kube-api-access-xcjhf\" (UniqueName: \"kubernetes.io/projected/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-kube-api-access-xcjhf\") pod \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\" (UID: \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\") " Nov 03 22:38:22.966949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.966842 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-plugins-config-volume\") pod \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\" (UID: \"8d5d8452-9a8e-4422-816c-e5ab42caa1bb\") " Nov 03 22:38:22.967335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.967309 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-plugins-config-volume" pod "8d5d8452-9a8e-4422-816c-e5ab42caa1bb" (UID: "8d5d8452-9a8e-4422-816c-e5ab42caa1bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:22.967718 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.967698 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:22.969109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:22.969077 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-kube-api-access-xcjhf" pod "8d5d8452-9a8e-4422-816c-e5ab42caa1bb" (UID: "8d5d8452-9a8e-4422-816c-e5ab42caa1bb"). InnerVolumeSpecName "kube-api-access-xcjhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:23.069287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.069215 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-kube-api-access-hbr92\" (UniqueName: \"kubernetes.io/projected/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-kube-api-access-hbr92\") pod \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\" (UID: \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\") " Nov 03 22:38:23.069476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.069467 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-plugins-config-volume\") pod \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\" (UID: \"152ffdb2-bb93-4293-a4ae-0ec17515b0d6\") " Nov 03 22:38:23.069835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.069807 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-plugins-config-volume" pod "152ffdb2-bb93-4293-a4ae-0ec17515b0d6" (UID: "152ffdb2-bb93-4293-a4ae-0ec17515b0d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:23.070255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.070232 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcjhf\" (UniqueName: \"kubernetes.io/projected/8d5d8452-9a8e-4422-816c-e5ab42caa1bb-kube-api-access-xcjhf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:23.070544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.070260 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:23.071619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.071595 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-kube-api-access-hbr92" pod "152ffdb2-bb93-4293-a4ae-0ec17515b0d6" (UID: "152ffdb2-bb93-4293-a4ae-0ec17515b0d6"). InnerVolumeSpecName "kube-api-access-hbr92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:23.171930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.171895 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbr92\" (UniqueName: \"kubernetes.io/projected/152ffdb2-bb93-4293-a4ae-0ec17515b0d6-kube-api-access-hbr92\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:23.282224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.282190 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct"] Nov 03 22:38:23.289330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.289302 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8b1efeb6-5d7dd8ff94-5blct"] Nov 03 22:38:23.736649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.736616 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5d8452-9a8e-4422-816c-e5ab42caa1bb" path="/var/lib/kubelet/pods/8d5d8452-9a8e-4422-816c-e5ab42caa1bb/volumes" Nov 03 22:38:23.944491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:23.944460 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz" Nov 03 22:38:24.015017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:24.014934 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz"] Nov 03 22:38:24.019250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:24.019225 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8b1efeb6-68787c86c9-5wqpz"] Nov 03 22:38:25.302696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:25.302662 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6"] Nov 03 22:38:25.303112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:25.302857 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerName="istio-proxy" containerID="cri-o://bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d" gracePeriod=30 Nov 03 22:38:25.737449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:25.737412 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152ffdb2-bb93-4293-a4ae-0ec17515b0d6" path="/var/lib/kubelet/pods/152ffdb2-bb93-4293-a4ae-0ec17515b0d6/volumes" Nov 03 22:38:26.426854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.426818 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm"] Nov 03 22:38:26.427792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.427773 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8d5d8452-9a8e-4422-816c-e5ab42caa1bb" containerName="payload-processing" Nov 03 22:38:26.427854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.427793 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5d8452-9a8e-4422-816c-e5ab42caa1bb" containerName="payload-processing" Nov 03 22:38:26.427854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.427825 4903 cpu_manager.go:414] "containerMap: removing container" podUID="152ffdb2-bb93-4293-a4ae-0ec17515b0d6" containerName="payload-pre-processing" Nov 03 22:38:26.427854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.427831 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152ffdb2-bb93-4293-a4ae-0ec17515b0d6" containerName="payload-pre-processing" Nov 03 22:38:26.427974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.427965 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5d8452-9a8e-4422-816c-e5ab42caa1bb" containerName="payload-processing" Nov 03 22:38:26.428010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.427982 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="152ffdb2-bb93-4293-a4ae-0ec17515b0d6" containerName="payload-pre-processing" Nov 03 22:38:26.433371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.433351 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:26.448214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.448184 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b1ec3a\"" Nov 03 22:38:26.450441 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.450414 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm"] Nov 03 22:38:26.478350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.478317 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx"] Nov 03 22:38:26.483236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.483212 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:26.497655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.497614 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b1ec3a-dockercfg-zt5cl\"" Nov 03 22:38:26.503996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.503965 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx"] Nov 03 22:38:26.538787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.538361 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xs5l\" (UniqueName: \"kubernetes.io/projected/8ac290e6-c72c-4184-9e06-e2e9af77eac2-kube-api-access-6xs5l\") pod \"payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm\" (UID: \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:26.538787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.538583 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac290e6-c72c-4184-9e06-e2e9af77eac2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm\" (UID: \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:26.650633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.649775 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xs5l\" (UniqueName: \"kubernetes.io/projected/8ac290e6-c72c-4184-9e06-e2e9af77eac2-kube-api-access-6xs5l\") pod \"payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm\" (UID: \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:26.650633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.649945 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf0936b-4b43-4543-9787-8ef2de2ae70f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx\" (UID: \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:26.650633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.650042 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac290e6-c72c-4184-9e06-e2e9af77eac2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm\" (UID: \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:26.650633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.650151 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjjdt\" (UniqueName: \"kubernetes.io/projected/9cf0936b-4b43-4543-9787-8ef2de2ae70f-kube-api-access-hjjdt\") pod \"payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx\" (UID: \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:26.673767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.673729 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac290e6-c72c-4184-9e06-e2e9af77eac2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm\" (UID: \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:26.704547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.704287 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xs5l\" (UniqueName: \"kubernetes.io/projected/8ac290e6-c72c-4184-9e06-e2e9af77eac2-kube-api-access-6xs5l\") pod \"payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm\" (UID: \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:26.762316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.756584 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:26.777485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.777449 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf0936b-4b43-4543-9787-8ef2de2ae70f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx\" (UID: \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:26.777649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.777633 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjjdt\" (UniqueName: \"kubernetes.io/projected/9cf0936b-4b43-4543-9787-8ef2de2ae70f-kube-api-access-hjjdt\") pod \"payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx\" (UID: \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:26.778526 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.778501 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf0936b-4b43-4543-9787-8ef2de2ae70f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx\" (UID: \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:26.799561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.799528 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjjdt\" (UniqueName: \"kubernetes.io/projected/9cf0936b-4b43-4543-9787-8ef2de2ae70f-kube-api-access-hjjdt\") pod \"payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx\" (UID: \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:26.802268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:26.802242 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:27.121253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:27.114760 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm"] Nov 03 22:38:27.154913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:27.154886 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx"] Nov 03 22:38:27.161791 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:38:27.161762 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf0936b_4b43_4543_9787_8ef2de2ae70f.slice/crio-a38443fdc585820dcfdff527dab718b6c85fdb01c1d785679781ac0c4aeba425 WatchSource:0}: Error finding container a38443fdc585820dcfdff527dab718b6c85fdb01c1d785679781ac0c4aeba425: Status 404 returned error can't find the container with id a38443fdc585820dcfdff527dab718b6c85fdb01c1d785679781ac0c4aeba425 Nov 03 22:38:27.987998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:27.987957 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" event={"ID":"9cf0936b-4b43-4543-9787-8ef2de2ae70f","Type":"ContainerStarted","Data":"8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7"} Nov 03 22:38:27.988510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:27.988026 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" event={"ID":"9cf0936b-4b43-4543-9787-8ef2de2ae70f","Type":"ContainerStarted","Data":"a38443fdc585820dcfdff527dab718b6c85fdb01c1d785679781ac0c4aeba425"} Nov 03 22:38:27.988510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:27.988107 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:27.990101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:27.990076 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" event={"ID":"8ac290e6-c72c-4184-9e06-e2e9af77eac2","Type":"ContainerStarted","Data":"2886b5f17189d074f6c7b145741ff805d6005b5bd83575404620f76fec86323e"} Nov 03 22:38:27.990250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:27.990132 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:27.990250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:27.990150 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" event={"ID":"8ac290e6-c72c-4184-9e06-e2e9af77eac2","Type":"ContainerStarted","Data":"c624fc8bb038d384f391d67d44c72e984c0f36688f5dfa8df9b658f917257a16"} Nov 03 22:38:28.066905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:28.066850 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" podStartSLOduration=2.066832357 podStartE2EDuration="2.066832357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:28.026598511 +0000 UTC m=+1534.430539486" watchObservedRunningTime="2026-11-03 22:38:28.066832357 +0000 UTC m=+1534.470773333" Nov 03 22:38:28.067671 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:28.067639 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" podStartSLOduration=2.067627904 podStartE2EDuration="2.067627904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:28.062226125 +0000 UTC m=+1534.466167105" watchObservedRunningTime="2026-11-03 22:38:28.067627904 +0000 UTC m=+1534.471568878" Nov 03 22:38:30.608549 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.608524 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:38:30.769237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.769201 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/480b3f78-d76d-4677-8d92-8fc048cb9185-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b3f78-d76d-4677-8d92-8fc048cb9185-istiod-ca-cert\") pod \"480b3f78-d76d-4677-8d92-8fc048cb9185\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " Nov 03 22:38:30.769615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.769587 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-credential-socket" pod "480b3f78-d76d-4677-8d92-8fc048cb9185" (UID: "480b3f78-d76d-4677-8d92-8fc048cb9185"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:30.769786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.769287 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-credential-socket\") pod \"480b3f78-d76d-4677-8d92-8fc048cb9185\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " Nov 03 22:38:30.769786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.769745 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-certs\") pod \"480b3f78-d76d-4677-8d92-8fc048cb9185\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " Nov 03 22:38:30.769786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.769767 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480b3f78-d76d-4677-8d92-8fc048cb9185-istiod-ca-cert" pod "480b3f78-d76d-4677-8d92-8fc048cb9185" (UID: "480b3f78-d76d-4677-8d92-8fc048cb9185"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:30.770013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.769842 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-data\") pod \"480b3f78-d76d-4677-8d92-8fc048cb9185\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " Nov 03 22:38:30.770013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.769906 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-token\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-token\") pod \"480b3f78-d76d-4677-8d92-8fc048cb9185\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " Nov 03 22:38:30.770013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.769988 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-envoy\") pod \"480b3f78-d76d-4677-8d92-8fc048cb9185\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " Nov 03 22:38:30.770205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.770029 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-kube-api-access-skcqt\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-kube-api-access-skcqt\") pod \"480b3f78-d76d-4677-8d92-8fc048cb9185\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " Nov 03 22:38:30.770205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.770155 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-data" pod "480b3f78-d76d-4677-8d92-8fc048cb9185" (UID: "480b3f78-d76d-4677-8d92-8fc048cb9185"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:30.770205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.770188 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-podinfo\") pod \"480b3f78-d76d-4677-8d92-8fc048cb9185\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " Nov 03 22:38:30.770478 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.770238 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-socket\") pod \"480b3f78-d76d-4677-8d92-8fc048cb9185\" (UID: \"480b3f78-d76d-4677-8d92-8fc048cb9185\") " Nov 03 22:38:30.770793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.770749 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-certs" pod "480b3f78-d76d-4677-8d92-8fc048cb9185" (UID: "480b3f78-d76d-4677-8d92-8fc048cb9185"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:30.770998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.770981 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/480b3f78-d76d-4677-8d92-8fc048cb9185-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:30.771130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.771004 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:30.771130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.771020 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:30.771130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.771036 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:30.771464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.771316 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-socket" pod "480b3f78-d76d-4677-8d92-8fc048cb9185" (UID: "480b3f78-d76d-4677-8d92-8fc048cb9185"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:30.774392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.774351 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-envoy" pod "480b3f78-d76d-4677-8d92-8fc048cb9185" (UID: "480b3f78-d76d-4677-8d92-8fc048cb9185"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:30.777161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.777122 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-kube-api-access-skcqt" pod "480b3f78-d76d-4677-8d92-8fc048cb9185" (UID: "480b3f78-d76d-4677-8d92-8fc048cb9185"). InnerVolumeSpecName "kube-api-access-skcqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:30.792937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.792599 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-podinfo" pod "480b3f78-d76d-4677-8d92-8fc048cb9185" (UID: "480b3f78-d76d-4677-8d92-8fc048cb9185"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:38:30.795648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.795597 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-token" pod "480b3f78-d76d-4677-8d92-8fc048cb9185" (UID: "480b3f78-d76d-4677-8d92-8fc048cb9185"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:30.895547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.875114 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:30.895899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.895876 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:30.896099 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.896063 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skcqt\" (UniqueName: \"kubernetes.io/projected/480b3f78-d76d-4677-8d92-8fc048cb9185-kube-api-access-skcqt\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:30.896224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.896210 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/480b3f78-d76d-4677-8d92-8fc048cb9185-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:30.896436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:30.896389 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/480b3f78-d76d-4677-8d92-8fc048cb9185-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:31.022939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.022888 4903 generic.go:356] "Generic (PLEG): container finished" podID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerID="bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d" exitCode=0 Nov 03 22:38:31.023148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.023048 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" event={"ID":"480b3f78-d76d-4677-8d92-8fc048cb9185","Type":"ContainerDied","Data":"bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d"} Nov 03 22:38:31.023148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.023105 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" event={"ID":"480b3f78-d76d-4677-8d92-8fc048cb9185","Type":"ContainerDied","Data":"c342ac4035c2c040d9e40cf84650bfea94bde896d37b2f768c37bcf10da0847b"} Nov 03 22:38:31.023272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.023139 4903 scope.go:122] "RemoveContainer" containerID="bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d" Nov 03 22:38:31.023378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.023315 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6" Nov 03 22:38:31.061389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.061359 4903 scope.go:122] "RemoveContainer" containerID="bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d" Nov 03 22:38:31.061799 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:31.061760 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d\": container with ID starting with bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d not found: ID does not exist" containerID="bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d" Nov 03 22:38:31.061928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.061802 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d"} err="failed to get container status \"bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d\": rpc error: code = NotFound desc = could not find container \"bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d\": container with ID starting with bdd61b57ab3a2331332ccc9b5d5feb58005a183dc14ac852e0d2c8c0aabca26d not found: ID does not exist" Nov 03 22:38:31.095948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.095911 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6"] Nov 03 22:38:31.104613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.104569 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-db8c6"] Nov 03 22:38:31.740173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:31.740136 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" path="/var/lib/kubelet/pods/480b3f78-d76d-4677-8d92-8fc048cb9185/volumes" Nov 03 22:38:32.851387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.851353 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2"] Nov 03 22:38:32.852638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.852620 4903 cpu_manager.go:414] "containerMap: removing container" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerName="istio-proxy" Nov 03 22:38:32.852701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.852641 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerName="istio-proxy" Nov 03 22:38:32.852872 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.852860 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="480b3f78-d76d-4677-8d92-8fc048cb9185" containerName="istio-proxy" Nov 03 22:38:32.859388 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.859361 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:32.877968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.877944 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2"] Nov 03 22:38:32.879344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.879283 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4b5e2b\"/\"test-model-4b5e2b-kserve-self-signed-certs\"" Nov 03 22:38:32.879519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.879499 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4b5e2b\"/\"default-dockercfg-rhk9d\"" Nov 03 22:38:32.947394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.947362 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4b5e2b\"/\"kube-root-ca.crt\"" Nov 03 22:38:32.947602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.947457 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4b5e2b\"/\"openshift-service-ca.crt\"" Nov 03 22:38:32.950580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.950550 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bf741a-dbda-425a-94b3-27e46a7d1a94-tls-certs\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:32.950810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.950778 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-tmp-dir\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:32.950890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.950834 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-model-cache\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:32.950966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.950943 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-home\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:32.951229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.951210 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6crq\" (UniqueName: \"kubernetes.io/projected/40bf741a-dbda-425a-94b3-27e46a7d1a94-kube-api-access-z6crq\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:32.951451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:32.951433 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-dshm\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.058484 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.058441 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6crq\" (UniqueName: \"kubernetes.io/projected/40bf741a-dbda-425a-94b3-27e46a7d1a94-kube-api-access-z6crq\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.062395 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.062363 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-dshm\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.062806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.062782 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bf741a-dbda-425a-94b3-27e46a7d1a94-tls-certs\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.062953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.062937 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-tmp-dir\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.063096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.063080 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-model-cache\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.063207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.063191 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-home\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.064243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.064220 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-home\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.065588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.065529 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-model-cache\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.066369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.066347 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-tmp-dir\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.078905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.078879 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bf741a-dbda-425a-94b3-27e46a7d1a94-tls-certs\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.080931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.080861 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-dshm\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.114821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.114741 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6crq\" (UniqueName: \"kubernetes.io/projected/40bf741a-dbda-425a-94b3-27e46a7d1a94-kube-api-access-z6crq\") pod \"test-model-4b5e2b-kserve-774dff4587-kdzz2\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.197693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.197657 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:33.530480 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:38:33.530444 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40bf741a_dbda_425a_94b3_27e46a7d1a94.slice/crio-5aeabcdeac62f2207450d4801b7165142498cb8b9a8d0cde387305c2ee35465f WatchSource:0}: Error finding container 5aeabcdeac62f2207450d4801b7165142498cb8b9a8d0cde387305c2ee35465f: Status 404 returned error can't find the container with id 5aeabcdeac62f2207450d4801b7165142498cb8b9a8d0cde387305c2ee35465f Nov 03 22:38:33.530717 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:33.530695 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2"] Nov 03 22:38:34.066547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:34.066509 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" event={"ID":"40bf741a-dbda-425a-94b3-27e46a7d1a94","Type":"ContainerStarted","Data":"91a5e29516f16afe8da000f51724828c5652344cdbe8fd1e2ddcd554df46a4d4"} Nov 03 22:38:34.067026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:34.066563 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" event={"ID":"40bf741a-dbda-425a-94b3-27e46a7d1a94","Type":"ContainerStarted","Data":"5aeabcdeac62f2207450d4801b7165142498cb8b9a8d0cde387305c2ee35465f"} Nov 03 22:38:34.112105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:34.112044 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" podStartSLOduration=2.112029522 podStartE2EDuration="2.112029522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:34.108895581 +0000 UTC m=+1540.512836555" watchObservedRunningTime="2026-11-03 22:38:34.112029522 +0000 UTC m=+1540.515970498" Nov 03 22:38:34.199102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:34.199074 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:35.608428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:35.608393 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l"] Nov 03 22:38:35.617561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:35.617531 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b1efeb6-j5j5l"] Nov 03 22:38:35.738630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:35.738600 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ef30f9-6a5b-435a-b572-9cda64ff9d14" path="/var/lib/kubelet/pods/31ef30f9-6a5b-435a-b572-9cda64ff9d14/volumes" Nov 03 22:38:39.001624 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:39.001594 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:38:39.002042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:39.001670 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:38:40.524536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.524500 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm"] Nov 03 22:38:40.530623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.530601 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.549004 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.548975 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm"] Nov 03 22:38:40.551532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.551499 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:38:40.551532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.551523 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:38:40.590549 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.590521 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:40.590713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.590621 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:38:40.606652 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.606615 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c0223-jbggm\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.606782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.606759 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c0223-jbggm\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.606941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.606925 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj4k6\" (UniqueName: \"kubernetes.io/projected/3aba7049-e353-4a86-8699-c8cf726b4fc8-kube-api-access-bj4k6\") pod \"maas-api-revoke-keys-e2e-api-a-2c0223-jbggm\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.708083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.708049 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj4k6\" (UniqueName: \"kubernetes.io/projected/3aba7049-e353-4a86-8699-c8cf726b4fc8-kube-api-access-bj4k6\") pod \"maas-api-revoke-keys-e2e-api-a-2c0223-jbggm\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.708233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.708186 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c0223-jbggm\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.708280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.708264 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c0223-jbggm\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.709015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.708988 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c0223-jbggm\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.710818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.710790 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c0223-jbggm\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.739062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.739034 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj4k6\" (UniqueName: \"kubernetes.io/projected/3aba7049-e353-4a86-8699-c8cf726b4fc8-kube-api-access-bj4k6\") pod \"maas-api-revoke-keys-e2e-api-a-2c0223-jbggm\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:40.851327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:40.851235 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:41.072086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:41.072036 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq"] Nov 03 22:38:41.165145 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:38:41.165121 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aba7049_e353_4a86_8699_c8cf726b4fc8.slice/crio-f3e081f4140e709b23cf4efccba1893a2e38c9d01d20262d04892b671ec434af WatchSource:0}: Error finding container f3e081f4140e709b23cf4efccba1893a2e38c9d01d20262d04892b671ec434af: Status 404 returned error can't find the container with id f3e081f4140e709b23cf4efccba1893a2e38c9d01d20262d04892b671ec434af Nov 03 22:38:41.166448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:41.166417 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm"] Nov 03 22:38:42.159866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:42.159826 4903 generic.go:356] "Generic (PLEG): container finished" podID="3aba7049-e353-4a86-8699-c8cf726b4fc8" containerID="49a84705010cfb4aa65ac2ce0b1e637cc8e8cfd0df46ccbb7732edc4a5174614" exitCode=0 Nov 03 22:38:42.160515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:42.159921 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" event={"ID":"3aba7049-e353-4a86-8699-c8cf726b4fc8","Type":"ContainerDied","Data":"49a84705010cfb4aa65ac2ce0b1e637cc8e8cfd0df46ccbb7732edc4a5174614"} Nov 03 22:38:42.160515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:42.159989 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" event={"ID":"3aba7049-e353-4a86-8699-c8cf726b4fc8","Type":"ContainerStarted","Data":"f3e081f4140e709b23cf4efccba1893a2e38c9d01d20262d04892b671ec434af"} Nov 03 22:38:43.199099 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.199057 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:43.213446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.213415 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:43.361601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.361579 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:43.457811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.457711 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-xks-ca\") pod \"3aba7049-e353-4a86-8699-c8cf726b4fc8\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " Nov 03 22:38:43.458047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.458022 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-openshift-ca\") pod \"3aba7049-e353-4a86-8699-c8cf726b4fc8\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " Nov 03 22:38:43.458137 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.458113 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aba7049-e353-4a86-8699-c8cf726b4fc8-kube-api-access-bj4k6\" (UniqueName: \"kubernetes.io/projected/3aba7049-e353-4a86-8699-c8cf726b4fc8-kube-api-access-bj4k6\") pod \"3aba7049-e353-4a86-8699-c8cf726b4fc8\" (UID: \"3aba7049-e353-4a86-8699-c8cf726b4fc8\") " Nov 03 22:38:43.460051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.460021 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-openshift-ca" pod "3aba7049-e353-4a86-8699-c8cf726b4fc8" (UID: "3aba7049-e353-4a86-8699-c8cf726b4fc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:43.461206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.461179 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aba7049-e353-4a86-8699-c8cf726b4fc8-kube-api-access-bj4k6" pod "3aba7049-e353-4a86-8699-c8cf726b4fc8" (UID: "3aba7049-e353-4a86-8699-c8cf726b4fc8"). InnerVolumeSpecName "kube-api-access-bj4k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:43.461410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.461363 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-xks-ca" pod "3aba7049-e353-4a86-8699-c8cf726b4fc8" (UID: "3aba7049-e353-4a86-8699-c8cf726b4fc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:43.561039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.561007 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:43.561039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.561035 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aba7049-e353-4a86-8699-c8cf726b4fc8-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:43.561039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:43.561045 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj4k6\" (UniqueName: \"kubernetes.io/projected/3aba7049-e353-4a86-8699-c8cf726b4fc8-kube-api-access-bj4k6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:44.183310 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:44.179090 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" Nov 03 22:38:44.183310 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:44.179118 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm" event={"ID":"3aba7049-e353-4a86-8699-c8cf726b4fc8","Type":"ContainerDied","Data":"f3e081f4140e709b23cf4efccba1893a2e38c9d01d20262d04892b671ec434af"} Nov 03 22:38:44.183310 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:44.179180 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3e081f4140e709b23cf4efccba1893a2e38c9d01d20262d04892b671ec434af" Nov 03 22:38:44.191077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:44.191053 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:38:46.430278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:46.430188 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq"] Nov 03 22:38:46.452486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:46.452449 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-32fdf2-qkccq"] Nov 03 22:38:47.737606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:47.737566 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38099760-6501-4a66-9a2d-8a1796e0eaee" path="/var/lib/kubelet/pods/38099760-6501-4a66-9a2d-8a1796e0eaee/volumes" Nov 03 22:38:48.385550 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:38:48.385510 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:38:51.053886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:51.053854 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq"] Nov 03 22:38:51.084502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:51.084469 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv"] Nov 03 22:38:54.828822 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.828791 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj"] Nov 03 22:38:54.829663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.829648 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3aba7049-e353-4a86-8699-c8cf726b4fc8" containerName="revoke-keys" Nov 03 22:38:54.829715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.829664 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aba7049-e353-4a86-8699-c8cf726b4fc8" containerName="revoke-keys" Nov 03 22:38:54.829830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.829820 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aba7049-e353-4a86-8699-c8cf726b4fc8" containerName="revoke-keys" Nov 03 22:38:54.835003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.834985 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:54.856459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.856226 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:38:54.857241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.857213 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:38:54.858142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.858115 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj"] Nov 03 22:38:54.858616 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.858587 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:38:54.858744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.858636 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:38:54.962420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.962383 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:54.962592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.962442 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:54.962645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:54.962623 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnl6r\" (UniqueName: \"kubernetes.io/projected/810ad376-f536-45ee-9624-1706cca2f59a-kube-api-access-rnl6r\") pod \"maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:55.063785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:55.063752 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnl6r\" (UniqueName: \"kubernetes.io/projected/810ad376-f536-45ee-9624-1706cca2f59a-kube-api-access-rnl6r\") pod \"maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:55.063977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:55.063959 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:55.064046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:55.064001 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:55.064667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:55.064640 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:55.066503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:55.066479 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:55.085669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:55.085612 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnl6r\" (UniqueName: \"kubernetes.io/projected/810ad376-f536-45ee-9624-1706cca2f59a-kube-api-access-rnl6r\") pod \"maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:55.167145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:55.167114 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:55.508717 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:38:55.508676 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810ad376_f536_45ee_9624_1706cca2f59a.slice/crio-bed5e9a7a3eb5db81fef3c5f2022b6933ec0472867d0ebd930adb1ebbe6cf1bf WatchSource:0}: Error finding container bed5e9a7a3eb5db81fef3c5f2022b6933ec0472867d0ebd930adb1ebbe6cf1bf: Status 404 returned error can't find the container with id bed5e9a7a3eb5db81fef3c5f2022b6933ec0472867d0ebd930adb1ebbe6cf1bf Nov 03 22:38:55.509156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:55.509135 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj"] Nov 03 22:38:56.055024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.054926 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" podUID="4511c67f-6f16-4586-92fc-74a8c4bf8b59" containerName="payload-processing" containerID="cri-o://8486530f7b9de35c956b730b9810c67610f231eda362edce245ec9ec305acde9" gracePeriod=25 Nov 03 22:38:56.084901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.084865 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" podUID="a3131409-8074-4901-8c15-3966d0c4b147" containerName="payload-pre-processing" containerID="cri-o://89dbcd66b36f14a6844d8c1d3b507072694873cfff359e913e904a41ec13b420" gracePeriod=25 Nov 03 22:38:56.096009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.095964 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" podUID="e7602f85-77a4-4ed5-991f-08ad57b49be3" containerName="main" containerID="cri-o://52379da38d9b289285f718d2c86827b6aa00ee3b9e643839f6d2018e7edfc54e" gracePeriod=45 Nov 03 22:38:56.104786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.104753 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" podUID="e7602f85-77a4-4ed5-991f-08ad57b49be3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.169:8000/ready\": dial tcp 10.132.0.169:8000: connect: connection refused" Nov 03 22:38:56.314223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.313879 4903 generic.go:356] "Generic (PLEG): container finished" podID="e7602f85-77a4-4ed5-991f-08ad57b49be3" containerID="52379da38d9b289285f718d2c86827b6aa00ee3b9e643839f6d2018e7edfc54e" exitCode=0 Nov 03 22:38:56.314223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.314062 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" event={"ID":"e7602f85-77a4-4ed5-991f-08ad57b49be3","Type":"ContainerDied","Data":"52379da38d9b289285f718d2c86827b6aa00ee3b9e643839f6d2018e7edfc54e"} Nov 03 22:38:56.316985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.316884 4903 generic.go:356] "Generic (PLEG): container finished" podID="a3131409-8074-4901-8c15-3966d0c4b147" containerID="89dbcd66b36f14a6844d8c1d3b507072694873cfff359e913e904a41ec13b420" exitCode=0 Nov 03 22:38:56.316985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.316949 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" event={"ID":"a3131409-8074-4901-8c15-3966d0c4b147","Type":"ContainerDied","Data":"89dbcd66b36f14a6844d8c1d3b507072694873cfff359e913e904a41ec13b420"} Nov 03 22:38:56.320333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.320203 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" event={"ID":"810ad376-f536-45ee-9624-1706cca2f59a","Type":"ContainerStarted","Data":"6ccf2f8664db98ff0516b3deca43d9077fbf27355e5a8da6daf60966e7f7695c"} Nov 03 22:38:56.320333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.320252 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" event={"ID":"810ad376-f536-45ee-9624-1706cca2f59a","Type":"ContainerStarted","Data":"bed5e9a7a3eb5db81fef3c5f2022b6933ec0472867d0ebd930adb1ebbe6cf1bf"} Nov 03 22:38:56.323329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.322694 4903 generic.go:356] "Generic (PLEG): container finished" podID="4511c67f-6f16-4586-92fc-74a8c4bf8b59" containerID="8486530f7b9de35c956b730b9810c67610f231eda362edce245ec9ec305acde9" exitCode=0 Nov 03 22:38:56.323329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.322731 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" event={"ID":"4511c67f-6f16-4586-92fc-74a8c4bf8b59","Type":"ContainerDied","Data":"8486530f7b9de35c956b730b9810c67610f231eda362edce245ec9ec305acde9"} Nov 03 22:38:56.371279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.371220 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" podStartSLOduration=2.371201268 podStartE2EDuration="2.371201268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:38:56.367398933 +0000 UTC m=+1562.771339910" watchObservedRunningTime="2026-11-03 22:38:56.371201268 +0000 UTC m=+1562.775142245" Nov 03 22:38:56.448344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.441273 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:38:56.514346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.513716 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:38:56.518492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.518465 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:38:56.595428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.595380 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3131409-8074-4901-8c15-3966d0c4b147-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3131409-8074-4901-8c15-3966d0c4b147-plugins-config-volume\") pod \"a3131409-8074-4901-8c15-3966d0c4b147\" (UID: \"a3131409-8074-4901-8c15-3966d0c4b147\") " Nov 03 22:38:56.595612 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.595576 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7602f85-77a4-4ed5-991f-08ad57b49be3-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7602f85-77a4-4ed5-991f-08ad57b49be3-tls-certs\") pod \"e7602f85-77a4-4ed5-991f-08ad57b49be3\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " Nov 03 22:38:56.595747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.595728 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-model-cache\") pod \"e7602f85-77a4-4ed5-991f-08ad57b49be3\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " Nov 03 22:38:56.595823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.595806 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-home\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-home\") pod \"e7602f85-77a4-4ed5-991f-08ad57b49be3\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " Nov 03 22:38:56.595878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.595845 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3131409-8074-4901-8c15-3966d0c4b147-plugins-config-volume" pod "a3131409-8074-4901-8c15-3966d0c4b147" (UID: "a3131409-8074-4901-8c15-3966d0c4b147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:56.595878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.595870 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7602f85-77a4-4ed5-991f-08ad57b49be3-kube-api-access-nnxrm\" (UniqueName: \"kubernetes.io/projected/e7602f85-77a4-4ed5-991f-08ad57b49be3-kube-api-access-nnxrm\") pod \"e7602f85-77a4-4ed5-991f-08ad57b49be3\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " Nov 03 22:38:56.595995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.595947 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3131409-8074-4901-8c15-3966d0c4b147-kube-api-access-v8h52\" (UniqueName: \"kubernetes.io/projected/a3131409-8074-4901-8c15-3966d0c4b147-kube-api-access-v8h52\") pod \"a3131409-8074-4901-8c15-3966d0c4b147\" (UID: \"a3131409-8074-4901-8c15-3966d0c4b147\") " Nov 03 22:38:56.595995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.595977 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4511c67f-6f16-4586-92fc-74a8c4bf8b59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4511c67f-6f16-4586-92fc-74a8c4bf8b59-plugins-config-volume\") pod \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\" (UID: \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\") " Nov 03 22:38:56.596092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.596059 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-tmp-dir\") pod \"e7602f85-77a4-4ed5-991f-08ad57b49be3\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " Nov 03 22:38:56.596146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.596088 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4511c67f-6f16-4586-92fc-74a8c4bf8b59-kube-api-access-62grr\" (UniqueName: \"kubernetes.io/projected/4511c67f-6f16-4586-92fc-74a8c4bf8b59-kube-api-access-62grr\") pod \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\" (UID: \"4511c67f-6f16-4586-92fc-74a8c4bf8b59\") " Nov 03 22:38:56.596204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.596166 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-dshm\") pod \"e7602f85-77a4-4ed5-991f-08ad57b49be3\" (UID: \"e7602f85-77a4-4ed5-991f-08ad57b49be3\") " Nov 03 22:38:56.596259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.596235 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-model-cache" pod "e7602f85-77a4-4ed5-991f-08ad57b49be3" (UID: "e7602f85-77a4-4ed5-991f-08ad57b49be3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:56.597017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.596991 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3131409-8074-4901-8c15-3966d0c4b147-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:56.597017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.597022 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:56.598326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.597668 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-home" pod "e7602f85-77a4-4ed5-991f-08ad57b49be3" (UID: "e7602f85-77a4-4ed5-991f-08ad57b49be3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:56.598326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.597592 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4511c67f-6f16-4586-92fc-74a8c4bf8b59-plugins-config-volume" pod "4511c67f-6f16-4586-92fc-74a8c4bf8b59" (UID: "4511c67f-6f16-4586-92fc-74a8c4bf8b59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:56.598326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.597824 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-tmp-dir" pod "e7602f85-77a4-4ed5-991f-08ad57b49be3" (UID: "e7602f85-77a4-4ed5-991f-08ad57b49be3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:56.599526 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.599498 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7602f85-77a4-4ed5-991f-08ad57b49be3-tls-certs" pod "e7602f85-77a4-4ed5-991f-08ad57b49be3" (UID: "e7602f85-77a4-4ed5-991f-08ad57b49be3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:56.600136 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.600109 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-dshm" pod "e7602f85-77a4-4ed5-991f-08ad57b49be3" (UID: "e7602f85-77a4-4ed5-991f-08ad57b49be3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:38:56.600965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.600932 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3131409-8074-4901-8c15-3966d0c4b147-kube-api-access-v8h52" pod "a3131409-8074-4901-8c15-3966d0c4b147" (UID: "a3131409-8074-4901-8c15-3966d0c4b147"). InnerVolumeSpecName "kube-api-access-v8h52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:56.601231 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.601208 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7602f85-77a4-4ed5-991f-08ad57b49be3-kube-api-access-nnxrm" pod "e7602f85-77a4-4ed5-991f-08ad57b49be3" (UID: "e7602f85-77a4-4ed5-991f-08ad57b49be3"). InnerVolumeSpecName "kube-api-access-nnxrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:56.601532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.601510 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4511c67f-6f16-4586-92fc-74a8c4bf8b59-kube-api-access-62grr" pod "4511c67f-6f16-4586-92fc-74a8c4bf8b59" (UID: "4511c67f-6f16-4586-92fc-74a8c4bf8b59"). InnerVolumeSpecName "kube-api-access-62grr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:56.700479 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.700440 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8h52\" (UniqueName: \"kubernetes.io/projected/a3131409-8074-4901-8c15-3966d0c4b147-kube-api-access-v8h52\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:56.700479 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.700479 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4511c67f-6f16-4586-92fc-74a8c4bf8b59-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:56.700745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.700493 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:56.700745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.700512 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62grr\" (UniqueName: \"kubernetes.io/projected/4511c67f-6f16-4586-92fc-74a8c4bf8b59-kube-api-access-62grr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:56.700745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.700524 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:56.700745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.700539 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7602f85-77a4-4ed5-991f-08ad57b49be3-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:56.700745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.700551 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7602f85-77a4-4ed5-991f-08ad57b49be3-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:56.700745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:56.700564 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxrm\" (UniqueName: \"kubernetes.io/projected/e7602f85-77a4-4ed5-991f-08ad57b49be3-kube-api-access-nnxrm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:57.340177 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.340147 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" Nov 03 22:38:57.340177 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.340153 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv" event={"ID":"a3131409-8074-4901-8c15-3966d0c4b147","Type":"ContainerDied","Data":"c109c4d5f0f2406a02b89c92d154558dbc26b045dfdaeeef8636b5666497a8a3"} Nov 03 22:38:57.340748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.340238 4903 scope.go:122] "RemoveContainer" containerID="89dbcd66b36f14a6844d8c1d3b507072694873cfff359e913e904a41ec13b420" Nov 03 22:38:57.342600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.342569 4903 generic.go:356] "Generic (PLEG): container finished" podID="810ad376-f536-45ee-9624-1706cca2f59a" containerID="6ccf2f8664db98ff0516b3deca43d9077fbf27355e5a8da6daf60966e7f7695c" exitCode=0 Nov 03 22:38:57.343001 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.342949 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" event={"ID":"810ad376-f536-45ee-9624-1706cca2f59a","Type":"ContainerDied","Data":"6ccf2f8664db98ff0516b3deca43d9077fbf27355e5a8da6daf60966e7f7695c"} Nov 03 22:38:57.344684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.344623 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" Nov 03 22:38:57.344913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.344675 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq" event={"ID":"4511c67f-6f16-4586-92fc-74a8c4bf8b59","Type":"ContainerDied","Data":"dff5ebbdd69b4b8b8e2587f12a29fe7d3d084ece170c4c9d4d5232b923b21826"} Nov 03 22:38:57.347592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.347570 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" event={"ID":"e7602f85-77a4-4ed5-991f-08ad57b49be3","Type":"ContainerDied","Data":"cea7c340bbf3792aedd3b9a69d7239611f9e46e118614525f4a901b37ea180e0"} Nov 03 22:38:57.347736 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.347584 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq" Nov 03 22:38:57.359868 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.359651 4903 scope.go:122] "RemoveContainer" containerID="8486530f7b9de35c956b730b9810c67610f231eda362edce245ec9ec305acde9" Nov 03 22:38:57.391741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.391710 4903 scope.go:122] "RemoveContainer" containerID="52379da38d9b289285f718d2c86827b6aa00ee3b9e643839f6d2018e7edfc54e" Nov 03 22:38:57.498113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.498073 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq"] Nov 03 22:38:57.508801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.508764 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c0223-696c7b845d-txtnq"] Nov 03 22:38:57.573199 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.573160 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv"] Nov 03 22:38:57.578095 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.578068 4903 scope.go:122] "RemoveContainer" containerID="12b38477bab24acec66949a4b5ef37600a8c5b1c1caa99945015e86f1cb9d21d" Nov 03 22:38:57.599030 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.598494 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c0223-5954dfbb87-n5vnv"] Nov 03 22:38:57.601134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.601110 4903 scope.go:122] "RemoveContainer" containerID="b8ccb680088ecb097ab62b9e2090cfc8aa543b680478c8ff3f00e67c23dd6817" Nov 03 22:38:57.614016 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.613985 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv"] Nov 03 22:38:57.647402 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.642494 4903 scope.go:122] "RemoveContainer" containerID="b4cc6679772b80a630b979ea378b76709e2fadf6dd69075d0cf87fb2bc3c24b8" Nov 03 22:38:57.647786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.647734 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bbfc2e-xxcqv"] Nov 03 22:38:57.669728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.669693 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq"] Nov 03 22:38:57.681991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.681953 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8b1efeb6/e2e-lifecycle-model-8b1efeb6-kserve-6fc5b9b7c7-779wq"] Nov 03 22:38:57.694912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.693468 4903 scope.go:122] "RemoveContainer" containerID="cbf88e6852ffda45bfe34fb22483fa56c9df353ddd4dbbe91432815ebc958aef" Nov 03 22:38:57.746646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.744507 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3852f9-c306-4bb0-862f-cbc9914f1daf" path="/var/lib/kubelet/pods/0e3852f9-c306-4bb0-862f-cbc9914f1daf/volumes" Nov 03 22:38:57.746646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.746203 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4511c67f-6f16-4586-92fc-74a8c4bf8b59" path="/var/lib/kubelet/pods/4511c67f-6f16-4586-92fc-74a8c4bf8b59/volumes" Nov 03 22:38:57.747140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.746948 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3131409-8074-4901-8c15-3966d0c4b147" path="/var/lib/kubelet/pods/a3131409-8074-4901-8c15-3966d0c4b147/volumes" Nov 03 22:38:57.747704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:57.747680 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7602f85-77a4-4ed5-991f-08ad57b49be3" path="/var/lib/kubelet/pods/e7602f85-77a4-4ed5-991f-08ad57b49be3/volumes" Nov 03 22:38:58.555058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.555033 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:58.652384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.652356 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-xks-ca\") pod \"810ad376-f536-45ee-9624-1706cca2f59a\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " Nov 03 22:38:58.652577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.652546 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810ad376-f536-45ee-9624-1706cca2f59a-kube-api-access-rnl6r\" (UniqueName: \"kubernetes.io/projected/810ad376-f536-45ee-9624-1706cca2f59a-kube-api-access-rnl6r\") pod \"810ad376-f536-45ee-9624-1706cca2f59a\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " Nov 03 22:38:58.652724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.652681 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-openshift-ca\") pod \"810ad376-f536-45ee-9624-1706cca2f59a\" (UID: \"810ad376-f536-45ee-9624-1706cca2f59a\") " Nov 03 22:38:58.653146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.653117 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-openshift-ca" pod "810ad376-f536-45ee-9624-1706cca2f59a" (UID: "810ad376-f536-45ee-9624-1706cca2f59a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:38:58.653586 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.653566 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:58.655110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.655086 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-xks-ca" pod "810ad376-f536-45ee-9624-1706cca2f59a" (UID: "810ad376-f536-45ee-9624-1706cca2f59a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:38:58.655221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.655205 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810ad376-f536-45ee-9624-1706cca2f59a-kube-api-access-rnl6r" pod "810ad376-f536-45ee-9624-1706cca2f59a" (UID: "810ad376-f536-45ee-9624-1706cca2f59a"). InnerVolumeSpecName "kube-api-access-rnl6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:38:58.754713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.754681 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnl6r\" (UniqueName: \"kubernetes.io/projected/810ad376-f536-45ee-9624-1706cca2f59a-kube-api-access-rnl6r\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:58.754713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:58.754710 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/810ad376-f536-45ee-9624-1706cca2f59a-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:38:59.367516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:59.367485 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" Nov 03 22:38:59.367673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:59.367518 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj" event={"ID":"810ad376-f536-45ee-9624-1706cca2f59a","Type":"ContainerDied","Data":"bed5e9a7a3eb5db81fef3c5f2022b6933ec0472867d0ebd930adb1ebbe6cf1bf"} Nov 03 22:38:59.367673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:38:59.367563 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed5e9a7a3eb5db81fef3c5f2022b6933ec0472867d0ebd930adb1ebbe6cf1bf" Nov 03 22:39:03.597826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.597792 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7"] Nov 03 22:39:03.598882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.598860 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a3131409-8074-4901-8c15-3966d0c4b147" containerName="payload-pre-processing" Nov 03 22:39:03.599014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.598883 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3131409-8074-4901-8c15-3966d0c4b147" containerName="payload-pre-processing" Nov 03 22:39:03.599014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.598925 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e7602f85-77a4-4ed5-991f-08ad57b49be3" containerName="main" Nov 03 22:39:03.599014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.598933 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7602f85-77a4-4ed5-991f-08ad57b49be3" containerName="main" Nov 03 22:39:03.599014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.598965 4903 cpu_manager.go:414] "containerMap: removing container" podUID="810ad376-f536-45ee-9624-1706cca2f59a" containerName="revoke-keys" Nov 03 22:39:03.599014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.598972 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810ad376-f536-45ee-9624-1706cca2f59a" containerName="revoke-keys" Nov 03 22:39:03.599014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.598984 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4511c67f-6f16-4586-92fc-74a8c4bf8b59" containerName="payload-processing" Nov 03 22:39:03.599014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.598990 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4511c67f-6f16-4586-92fc-74a8c4bf8b59" containerName="payload-processing" Nov 03 22:39:03.599425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.599211 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3131409-8074-4901-8c15-3966d0c4b147" containerName="payload-pre-processing" Nov 03 22:39:03.599425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.599228 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7602f85-77a4-4ed5-991f-08ad57b49be3" containerName="main" Nov 03 22:39:03.599425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.599243 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="810ad376-f536-45ee-9624-1706cca2f59a" containerName="revoke-keys" Nov 03 22:39:03.599425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.599253 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4511c67f-6f16-4586-92fc-74a8c4bf8b59" containerName="payload-processing" Nov 03 22:39:03.604819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.604798 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.627235 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.625873 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7"] Nov 03 22:39:03.631271 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.631242 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2d6076\"/\"test-model-2d6076-kserve-self-signed-certs\"" Nov 03 22:39:03.632161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.632139 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2d6076\"/\"openshift-service-ca.crt\"" Nov 03 22:39:03.632347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.632258 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2d6076\"/\"kube-root-ca.crt\"" Nov 03 22:39:03.632444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.632378 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2d6076\"/\"default-dockercfg-frlpl\"" Nov 03 22:39:03.655345 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.655312 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-model-cache\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.655573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.655543 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7nrz\" (UniqueName: \"kubernetes.io/projected/10347389-d227-4891-9a5c-401dec199b97-kube-api-access-c7nrz\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.655657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.655633 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-home\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.655884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.655696 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10347389-d227-4891-9a5c-401dec199b97-tls-certs\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.655884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.655835 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-dshm\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.655884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.655877 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-tmp-dir\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.757873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.757833 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7nrz\" (UniqueName: \"kubernetes.io/projected/10347389-d227-4891-9a5c-401dec199b97-kube-api-access-c7nrz\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.757873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.757875 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-home\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.758133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.757920 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10347389-d227-4891-9a5c-401dec199b97-tls-certs\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.758201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.758178 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-dshm\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.758261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.758221 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-tmp-dir\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.758357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.758329 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-home\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.758459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.758438 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-model-cache\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.758459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.758452 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-tmp-dir\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.758748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.758726 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-model-cache\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.760374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.760350 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-dshm\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.760700 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.760680 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10347389-d227-4891-9a5c-401dec199b97-tls-certs\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.783495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.783467 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7nrz\" (UniqueName: \"kubernetes.io/projected/10347389-d227-4891-9a5c-401dec199b97-kube-api-access-c7nrz\") pod \"test-model-2d6076-kserve-f99c557c-bdmv7\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:03.926645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:03.926608 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:04.268197 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:39:04.268162 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10347389_d227_4891_9a5c_401dec199b97.slice/crio-070e7303b9bd417b9989a77f18420c4338b46434f222b8ac644d04a158073b9b WatchSource:0}: Error finding container 070e7303b9bd417b9989a77f18420c4338b46434f222b8ac644d04a158073b9b: Status 404 returned error can't find the container with id 070e7303b9bd417b9989a77f18420c4338b46434f222b8ac644d04a158073b9b Nov 03 22:39:04.268623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:04.268601 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7"] Nov 03 22:39:04.412458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:04.412420 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" event={"ID":"10347389-d227-4891-9a5c-401dec199b97","Type":"ContainerStarted","Data":"129e43adaf286f4b5119f5e5d984982881cb09266b17bd89826f2d74839f2884"} Nov 03 22:39:04.412633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:04.412469 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" event={"ID":"10347389-d227-4891-9a5c-401dec199b97","Type":"ContainerStarted","Data":"070e7303b9bd417b9989a77f18420c4338b46434f222b8ac644d04a158073b9b"} Nov 03 22:39:04.455386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:04.455319 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" podStartSLOduration=1.455282064 podStartE2EDuration="1.455282064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:04.449760705 +0000 UTC m=+1570.853701681" watchObservedRunningTime="2026-11-03 22:39:04.455282064 +0000 UTC m=+1570.859223040" Nov 03 22:39:04.928281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:04.928245 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:05.335365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:05.334629 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw"] Nov 03 22:39:05.385884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:05.385844 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w"] Nov 03 22:39:10.335355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.335222 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" podUID="535c9b80-60fc-499c-a317-9cdac646d1ab" containerName="payload-processing" containerID="cri-o://1715be75365f0f8f396830f16d0f3c58f4403b4de615e4fba8ad9419a0b0345a" gracePeriod=25 Nov 03 22:39:10.386843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.386778 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" podUID="91fb5828-1e15-4e28-b558-6b4fd01920a8" containerName="payload-pre-processing" containerID="cri-o://b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae" gracePeriod=25 Nov 03 22:39:10.469632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.469602 4903 generic.go:356] "Generic (PLEG): container finished" podID="535c9b80-60fc-499c-a317-9cdac646d1ab" containerID="1715be75365f0f8f396830f16d0f3c58f4403b4de615e4fba8ad9419a0b0345a" exitCode=0 Nov 03 22:39:10.469880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.469853 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" event={"ID":"535c9b80-60fc-499c-a317-9cdac646d1ab","Type":"ContainerDied","Data":"1715be75365f0f8f396830f16d0f3c58f4403b4de615e4fba8ad9419a0b0345a"} Nov 03 22:39:10.678729 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.678696 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp"] Nov 03 22:39:10.687463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.686196 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65b0e2-tdbqp"] Nov 03 22:39:10.762340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.759661 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:39:10.866032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.865927 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535c9b80-60fc-499c-a317-9cdac646d1ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c9b80-60fc-499c-a317-9cdac646d1ab-plugins-config-volume\") pod \"535c9b80-60fc-499c-a317-9cdac646d1ab\" (UID: \"535c9b80-60fc-499c-a317-9cdac646d1ab\") " Nov 03 22:39:10.866032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.865994 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535c9b80-60fc-499c-a317-9cdac646d1ab-kube-api-access-z6cdw\" (UniqueName: \"kubernetes.io/projected/535c9b80-60fc-499c-a317-9cdac646d1ab-kube-api-access-z6cdw\") pod \"535c9b80-60fc-499c-a317-9cdac646d1ab\" (UID: \"535c9b80-60fc-499c-a317-9cdac646d1ab\") " Nov 03 22:39:10.869048 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.869013 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535c9b80-60fc-499c-a317-9cdac646d1ab-plugins-config-volume" pod "535c9b80-60fc-499c-a317-9cdac646d1ab" (UID: "535c9b80-60fc-499c-a317-9cdac646d1ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:10.871913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.871888 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535c9b80-60fc-499c-a317-9cdac646d1ab-kube-api-access-z6cdw" pod "535c9b80-60fc-499c-a317-9cdac646d1ab" (UID: "535c9b80-60fc-499c-a317-9cdac646d1ab"). InnerVolumeSpecName "kube-api-access-z6cdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:10.905962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.905933 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:39:10.970882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.970801 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c9b80-60fc-499c-a317-9cdac646d1ab-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:10.970882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:10.970841 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6cdw\" (UniqueName: \"kubernetes.io/projected/535c9b80-60fc-499c-a317-9cdac646d1ab-kube-api-access-z6cdw\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.071447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.071413 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fb5828-1e15-4e28-b558-6b4fd01920a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fb5828-1e15-4e28-b558-6b4fd01920a8-plugins-config-volume\") pod \"91fb5828-1e15-4e28-b558-6b4fd01920a8\" (UID: \"91fb5828-1e15-4e28-b558-6b4fd01920a8\") " Nov 03 22:39:11.071674 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.071658 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fb5828-1e15-4e28-b558-6b4fd01920a8-kube-api-access-cd5wr\" (UniqueName: \"kubernetes.io/projected/91fb5828-1e15-4e28-b558-6b4fd01920a8-kube-api-access-cd5wr\") pod \"91fb5828-1e15-4e28-b558-6b4fd01920a8\" (UID: \"91fb5828-1e15-4e28-b558-6b4fd01920a8\") " Nov 03 22:39:11.071851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.071823 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fb5828-1e15-4e28-b558-6b4fd01920a8-plugins-config-volume" pod "91fb5828-1e15-4e28-b558-6b4fd01920a8" (UID: "91fb5828-1e15-4e28-b558-6b4fd01920a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:11.072228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.072213 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fb5828-1e15-4e28-b558-6b4fd01920a8-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.073897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.073877 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fb5828-1e15-4e28-b558-6b4fd01920a8-kube-api-access-cd5wr" pod "91fb5828-1e15-4e28-b558-6b4fd01920a8" (UID: "91fb5828-1e15-4e28-b558-6b4fd01920a8"). InnerVolumeSpecName "kube-api-access-cd5wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:11.174408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.174374 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd5wr\" (UniqueName: \"kubernetes.io/projected/91fb5828-1e15-4e28-b558-6b4fd01920a8-kube-api-access-cd5wr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:11.485136 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.485103 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" Nov 03 22:39:11.485136 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.485128 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw" event={"ID":"535c9b80-60fc-499c-a317-9cdac646d1ab","Type":"ContainerDied","Data":"8f2aea74b24cd76ac82ba4e3da135d749fe607cdb0392adc19ce8abe172f7e83"} Nov 03 22:39:11.485678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.485205 4903 scope.go:122] "RemoveContainer" containerID="1715be75365f0f8f396830f16d0f3c58f4403b4de615e4fba8ad9419a0b0345a" Nov 03 22:39:11.486725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.486702 4903 generic.go:356] "Generic (PLEG): container finished" podID="91fb5828-1e15-4e28-b558-6b4fd01920a8" containerID="b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae" exitCode=0 Nov 03 22:39:11.486808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.486745 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" Nov 03 22:39:11.486808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.486743 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" event={"ID":"91fb5828-1e15-4e28-b558-6b4fd01920a8","Type":"ContainerDied","Data":"b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae"} Nov 03 22:39:11.486808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.486795 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w" event={"ID":"91fb5828-1e15-4e28-b558-6b4fd01920a8","Type":"ContainerDied","Data":"b5c0ddf0344db5c43200b06409db45d20d6df4ccf13a5b55eda4c424ad268d1b"} Nov 03 22:39:11.506204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.506078 4903 scope.go:122] "RemoveContainer" containerID="b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae" Nov 03 22:39:11.518184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.518160 4903 scope.go:122] "RemoveContainer" containerID="b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae" Nov 03 22:39:11.518480 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:39:11.518461 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae\": container with ID starting with b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae not found: ID does not exist" containerID="b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae" Nov 03 22:39:11.518539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.518489 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae"} err="failed to get container status \"b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae\": rpc error: code = NotFound desc = could not find container \"b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae\": container with ID starting with b5d5bd02f2866d74b7364487e8310f6f6fb17d044a756a242c0730ac40b3d7ae not found: ID does not exist" Nov 03 22:39:11.555621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.555591 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw"] Nov 03 22:39:11.559847 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.559827 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b826b6-65dbdbf768-v9wnw"] Nov 03 22:39:11.588848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.588816 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w"] Nov 03 22:39:11.593451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.593426 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b826b6-86b86bbbff-rkg2w"] Nov 03 22:39:11.740884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.740807 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="535c9b80-60fc-499c-a317-9cdac646d1ab" path="/var/lib/kubelet/pods/535c9b80-60fc-499c-a317-9cdac646d1ab/volumes" Nov 03 22:39:11.741330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.741312 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fb5828-1e15-4e28-b558-6b4fd01920a8" path="/var/lib/kubelet/pods/91fb5828-1e15-4e28-b558-6b4fd01920a8/volumes" Nov 03 22:39:11.741862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:11.741844 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89c212b-9dd9-426e-9bb1-4a0652035966" path="/var/lib/kubelet/pods/d89c212b-9dd9-426e-9bb1-4a0652035966/volumes" Nov 03 22:39:13.928341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:13.928260 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:13.941455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:13.941430 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:14.541767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:14.541736 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:39:16.104440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.104408 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz"] Nov 03 22:39:16.104833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.104637 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" containerID="cri-o://4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69" gracePeriod=30 Nov 03 22:39:16.875525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.875493 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q"] Nov 03 22:39:16.876418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.876395 4903 cpu_manager.go:414] "containerMap: removing container" podUID="535c9b80-60fc-499c-a317-9cdac646d1ab" containerName="payload-processing" Nov 03 22:39:16.876418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.876419 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535c9b80-60fc-499c-a317-9cdac646d1ab" containerName="payload-processing" Nov 03 22:39:16.876677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.876528 4903 cpu_manager.go:414] "containerMap: removing container" podUID="91fb5828-1e15-4e28-b558-6b4fd01920a8" containerName="payload-pre-processing" Nov 03 22:39:16.876677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.876538 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fb5828-1e15-4e28-b558-6b4fd01920a8" containerName="payload-pre-processing" Nov 03 22:39:16.876785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.876755 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fb5828-1e15-4e28-b558-6b4fd01920a8" containerName="payload-pre-processing" Nov 03 22:39:16.876785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.876773 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="535c9b80-60fc-499c-a317-9cdac646d1ab" containerName="payload-processing" Nov 03 22:39:16.880042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.880018 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.905926 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.905895 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9e540d-openshift-default-dockercfg-fj7p7\"" Nov 03 22:39:16.907278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.907250 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967c277a-6315-4353-8e39-b05adb456f1d-istiod-ca-cert\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.907392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.907334 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-credential-socket\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.907469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.907399 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-certs\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.907469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.907427 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8f4\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-kube-api-access-7s8f4\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.907469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.907454 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-data\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.907610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.907484 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-envoy\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.907610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.907506 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967c277a-6315-4353-8e39-b05adb456f1d-istio-podinfo\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.907610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.907548 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-socket\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.907743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.907624 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-istio-token\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:16.916340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:16.912816 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q"] Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.010743 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967c277a-6315-4353-8e39-b05adb456f1d-istiod-ca-cert\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.010843 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-credential-socket\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.010941 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-certs\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.010970 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8f4\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-kube-api-access-7s8f4\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.011000 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-data\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.011035 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-envoy\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.011061 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967c277a-6315-4353-8e39-b05adb456f1d-istio-podinfo\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.011118 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-socket\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.011209 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-istio-token\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.012096 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-data\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.012237 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967c277a-6315-4353-8e39-b05adb456f1d-istiod-ca-cert\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.012467 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-credential-socket\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.012657 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-certs\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.015939 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967c277a-6315-4353-8e39-b05adb456f1d-istio-podinfo\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.020650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.016225 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-socket\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.047370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.044917 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-envoy\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.081327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.079342 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8f4\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-kube-api-access-7s8f4\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.093315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.092856 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-istio-token\") pod \"e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.200494 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.199589 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:17.565113 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:39:17.565078 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967c277a_6315_4353_8e39_b05adb456f1d.slice/crio-2165aecb2fe1e9840527eb8974d17ee40e0b2037bf9500cf1c982b7cf03ce9d4 WatchSource:0}: Error finding container 2165aecb2fe1e9840527eb8974d17ee40e0b2037bf9500cf1c982b7cf03ce9d4: Status 404 returned error can't find the container with id 2165aecb2fe1e9840527eb8974d17ee40e0b2037bf9500cf1c982b7cf03ce9d4 Nov 03 22:39:17.565477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.565454 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q"] Nov 03 22:39:17.567998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.567968 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:17.568101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.568029 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:17.568101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:17.568058 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:18.387034 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:39:18.387001 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:39:18.577110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:18.577070 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" event={"ID":"967c277a-6315-4353-8e39-b05adb456f1d","Type":"ContainerStarted","Data":"8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0"} Nov 03 22:39:18.577286 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:18.577134 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" event={"ID":"967c277a-6315-4353-8e39-b05adb456f1d","Type":"ContainerStarted","Data":"2165aecb2fe1e9840527eb8974d17ee40e0b2037bf9500cf1c982b7cf03ce9d4"} Nov 03 22:39:18.621185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:18.621122 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" podStartSLOduration=2.621103912 podStartE2EDuration="2.621103912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:18.618632815 +0000 UTC m=+1585.022573791" watchObservedRunningTime="2026-11-03 22:39:18.621103912 +0000 UTC m=+1585.025044888" Nov 03 22:39:19.200178 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:19.200149 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:19.207126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:19.207100 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:19.588360 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:19.588269 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:19.589545 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:19.589519 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:39:20.198253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.198220 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz"] Nov 03 22:39:20.203989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.203962 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.219890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.219859 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz"] Nov 03 22:39:20.224029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.224005 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:39:20.224029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.224008 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:39:20.301570 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.301538 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzt9k\" (UniqueName: \"kubernetes.io/projected/2fca6c22-9dfb-49ec-8051-4dbd379742a8-kube-api-access-qzt9k\") pod \"maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.301756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.301604 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.301756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.301732 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.309020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.308994 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:39:20.309234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.309220 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:39:20.402915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.402880 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzt9k\" (UniqueName: \"kubernetes.io/projected/2fca6c22-9dfb-49ec-8051-4dbd379742a8-kube-api-access-qzt9k\") pod \"maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.403083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.402942 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.403083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.403006 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.403653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.403633 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.405308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.405274 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.425241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.425219 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzt9k\" (UniqueName: \"kubernetes.io/projected/2fca6c22-9dfb-49ec-8051-4dbd379742a8-kube-api-access-qzt9k\") pod \"maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.524019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.523938 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:20.836978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:20.836948 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz"] Nov 03 22:39:20.837358 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:39:20.836990 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fca6c22_9dfb_49ec_8051_4dbd379742a8.slice/crio-febdb58b15ac87eef593a8ac5aee7f9e3ab26f90704912627563757adfe1af1e WatchSource:0}: Error finding container febdb58b15ac87eef593a8ac5aee7f9e3ab26f90704912627563757adfe1af1e: Status 404 returned error can't find the container with id febdb58b15ac87eef593a8ac5aee7f9e3ab26f90704912627563757adfe1af1e Nov 03 22:39:21.377536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.377512 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:39:21.420554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.420525 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92980776-9c6e-48e9-ae02-4dbaecabc575-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92980776-9c6e-48e9-ae02-4dbaecabc575-istiod-ca-cert\") pod \"92980776-9c6e-48e9-ae02-4dbaecabc575\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " Nov 03 22:39:21.420772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.420754 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-certs\") pod \"92980776-9c6e-48e9-ae02-4dbaecabc575\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " Nov 03 22:39:21.420853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.420819 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-podinfo\") pod \"92980776-9c6e-48e9-ae02-4dbaecabc575\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " Nov 03 22:39:21.420908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.420863 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-data\") pod \"92980776-9c6e-48e9-ae02-4dbaecabc575\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " Nov 03 22:39:21.420963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.420923 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92980776-9c6e-48e9-ae02-4dbaecabc575-istiod-ca-cert" pod "92980776-9c6e-48e9-ae02-4dbaecabc575" (UID: "92980776-9c6e-48e9-ae02-4dbaecabc575"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:21.420963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.420934 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-token\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-token\") pod \"92980776-9c6e-48e9-ae02-4dbaecabc575\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " Nov 03 22:39:21.421143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.421098 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-certs" pod "92980776-9c6e-48e9-ae02-4dbaecabc575" (UID: "92980776-9c6e-48e9-ae02-4dbaecabc575"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:21.421209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.421164 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-data" pod "92980776-9c6e-48e9-ae02-4dbaecabc575" (UID: "92980776-9c6e-48e9-ae02-4dbaecabc575"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:21.421209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.421199 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-envoy\") pod \"92980776-9c6e-48e9-ae02-4dbaecabc575\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " Nov 03 22:39:21.421317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.421277 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-credential-socket\") pod \"92980776-9c6e-48e9-ae02-4dbaecabc575\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " Nov 03 22:39:21.421382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.421343 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-socket\") pod \"92980776-9c6e-48e9-ae02-4dbaecabc575\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " Nov 03 22:39:21.421442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.421386 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-kube-api-access-4pvkf\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-kube-api-access-4pvkf\") pod \"92980776-9c6e-48e9-ae02-4dbaecabc575\" (UID: \"92980776-9c6e-48e9-ae02-4dbaecabc575\") " Nov 03 22:39:21.421596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.421576 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-credential-socket" pod "92980776-9c6e-48e9-ae02-4dbaecabc575" (UID: "92980776-9c6e-48e9-ae02-4dbaecabc575"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:21.421694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.421656 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-socket" pod "92980776-9c6e-48e9-ae02-4dbaecabc575" (UID: "92980776-9c6e-48e9-ae02-4dbaecabc575"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:21.422198 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.422174 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:21.422288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.422205 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:21.422288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.422220 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:21.422288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.422236 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92980776-9c6e-48e9-ae02-4dbaecabc575-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:21.422288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.422249 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:21.423499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.423471 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-podinfo" pod "92980776-9c6e-48e9-ae02-4dbaecabc575" (UID: "92980776-9c6e-48e9-ae02-4dbaecabc575"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:21.423694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.423667 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-token" pod "92980776-9c6e-48e9-ae02-4dbaecabc575" (UID: "92980776-9c6e-48e9-ae02-4dbaecabc575"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:21.424232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.424203 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-envoy" pod "92980776-9c6e-48e9-ae02-4dbaecabc575" (UID: "92980776-9c6e-48e9-ae02-4dbaecabc575"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:21.424540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.424518 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-kube-api-access-4pvkf" pod "92980776-9c6e-48e9-ae02-4dbaecabc575" (UID: "92980776-9c6e-48e9-ae02-4dbaecabc575"). InnerVolumeSpecName "kube-api-access-4pvkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:21.524443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.524405 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:21.524443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.524442 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:21.524637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.524456 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92980776-9c6e-48e9-ae02-4dbaecabc575-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:21.524637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.524470 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pvkf\" (UniqueName: \"kubernetes.io/projected/92980776-9c6e-48e9-ae02-4dbaecabc575-kube-api-access-4pvkf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:21.611790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.611749 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" event={"ID":"2fca6c22-9dfb-49ec-8051-4dbd379742a8","Type":"ContainerStarted","Data":"bacae7d091abd4b7c06df688fc64b55197fd58841cf696e2427fa99cd0281baa"} Nov 03 22:39:21.612182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.611825 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" event={"ID":"2fca6c22-9dfb-49ec-8051-4dbd379742a8","Type":"ContainerStarted","Data":"febdb58b15ac87eef593a8ac5aee7f9e3ab26f90704912627563757adfe1af1e"} Nov 03 22:39:21.613424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.613394 4903 generic.go:356] "Generic (PLEG): container finished" podID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerID="4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69" exitCode=0 Nov 03 22:39:21.613702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.613447 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" Nov 03 22:39:21.613702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.613488 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" event={"ID":"92980776-9c6e-48e9-ae02-4dbaecabc575","Type":"ContainerDied","Data":"4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69"} Nov 03 22:39:21.613702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.613535 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz" event={"ID":"92980776-9c6e-48e9-ae02-4dbaecabc575","Type":"ContainerDied","Data":"fa3a5917e262f8310e731c1811c9c9d0f73e24f08d7830dac86d4239d885ad2a"} Nov 03 22:39:21.613702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.613563 4903 scope.go:122] "RemoveContainer" containerID="4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69" Nov 03 22:39:21.632177 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.632154 4903 scope.go:122] "RemoveContainer" containerID="4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69" Nov 03 22:39:21.632560 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:39:21.632536 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69\": container with ID starting with 4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69 not found: ID does not exist" containerID="4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69" Nov 03 22:39:21.632659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.632568 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69"} err="failed to get container status \"4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69\": rpc error: code = NotFound desc = could not find container \"4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69\": container with ID starting with 4f57f69aedac9ed027ffe13341675564a79cc592b390cb3d4665042dfe366a69 not found: ID does not exist" Nov 03 22:39:21.665785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.665720 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" podStartSLOduration=1.665702018 podStartE2EDuration="1.665702018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:21.663920905 +0000 UTC m=+1588.067861876" watchObservedRunningTime="2026-11-03 22:39:21.665702018 +0000 UTC m=+1588.069642994" Nov 03 22:39:21.713028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.712997 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz"] Nov 03 22:39:21.720086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.720054 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-6skbz"] Nov 03 22:39:21.738217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.738185 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" path="/var/lib/kubelet/pods/92980776-9c6e-48e9-ae02-4dbaecabc575/volumes" Nov 03 22:39:21.969316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:21.966409 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:39:22.625665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:22.625632 4903 generic.go:356] "Generic (PLEG): container finished" podID="2fca6c22-9dfb-49ec-8051-4dbd379742a8" containerID="bacae7d091abd4b7c06df688fc64b55197fd58841cf696e2427fa99cd0281baa" exitCode=0 Nov 03 22:39:22.625873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:22.625720 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" event={"ID":"2fca6c22-9dfb-49ec-8051-4dbd379742a8","Type":"ContainerDied","Data":"bacae7d091abd4b7c06df688fc64b55197fd58841cf696e2427fa99cd0281baa"} Nov 03 22:39:23.300626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.300592 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b"] Nov 03 22:39:23.301502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.301485 4903 cpu_manager.go:414] "containerMap: removing container" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" Nov 03 22:39:23.301562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.301504 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" Nov 03 22:39:23.301662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.301652 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="92980776-9c6e-48e9-ae02-4dbaecabc575" containerName="istio-proxy" Nov 03 22:39:23.305654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.305634 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:23.322369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.321649 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9e540d\"" Nov 03 22:39:23.390254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.390217 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9e540d-dockercfg-dsvmc\"" Nov 03 22:39:23.391395 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.391340 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b"] Nov 03 22:39:23.396814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.396783 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55e5d3f-2802-49a3-a784-602e3d1c1327-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b\" (UID: \"f55e5d3f-2802-49a3-a784-602e3d1c1327\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:23.397093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.397071 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rrk4\" (UniqueName: \"kubernetes.io/projected/f55e5d3f-2802-49a3-a784-602e3d1c1327-kube-api-access-7rrk4\") pod \"payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b\" (UID: \"f55e5d3f-2802-49a3-a784-602e3d1c1327\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:23.549315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.533709 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55e5d3f-2802-49a3-a784-602e3d1c1327-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b\" (UID: \"f55e5d3f-2802-49a3-a784-602e3d1c1327\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:23.549315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.534005 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrk4\" (UniqueName: \"kubernetes.io/projected/f55e5d3f-2802-49a3-a784-602e3d1c1327-kube-api-access-7rrk4\") pod \"payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b\" (UID: \"f55e5d3f-2802-49a3-a784-602e3d1c1327\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:23.549315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.536643 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55e5d3f-2802-49a3-a784-602e3d1c1327-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b\" (UID: \"f55e5d3f-2802-49a3-a784-602e3d1c1327\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:23.589350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.589252 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rrk4\" (UniqueName: \"kubernetes.io/projected/f55e5d3f-2802-49a3-a784-602e3d1c1327-kube-api-access-7rrk4\") pod \"payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b\" (UID: \"f55e5d3f-2802-49a3-a784-602e3d1c1327\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:23.623946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.623913 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:23.982441 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:23.982408 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:24.060675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.060644 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-openshift-ca\") pod \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " Nov 03 22:39:24.060872 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.060775 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-xks-ca\") pod \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " Nov 03 22:39:24.060872 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.060844 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fca6c22-9dfb-49ec-8051-4dbd379742a8-kube-api-access-qzt9k\" (UniqueName: \"kubernetes.io/projected/2fca6c22-9dfb-49ec-8051-4dbd379742a8-kube-api-access-qzt9k\") pod \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\" (UID: \"2fca6c22-9dfb-49ec-8051-4dbd379742a8\") " Nov 03 22:39:24.061369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.061053 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-openshift-ca" pod "2fca6c22-9dfb-49ec-8051-4dbd379742a8" (UID: "2fca6c22-9dfb-49ec-8051-4dbd379742a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:24.061723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.061701 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:24.063635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.063611 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-xks-ca" pod "2fca6c22-9dfb-49ec-8051-4dbd379742a8" (UID: "2fca6c22-9dfb-49ec-8051-4dbd379742a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:39:24.064221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.064190 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fca6c22-9dfb-49ec-8051-4dbd379742a8-kube-api-access-qzt9k" pod "2fca6c22-9dfb-49ec-8051-4dbd379742a8" (UID: "2fca6c22-9dfb-49ec-8051-4dbd379742a8"). InnerVolumeSpecName "kube-api-access-qzt9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:24.078661 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.078391 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b"] Nov 03 22:39:24.079375 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:39:24.079105 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55e5d3f_2802_49a3_a784_602e3d1c1327.slice/crio-ee98209422826f9d0986b591abaea2d257f1797818b9387c79687f827d00aafd WatchSource:0}: Error finding container ee98209422826f9d0986b591abaea2d257f1797818b9387c79687f827d00aafd: Status 404 returned error can't find the container with id ee98209422826f9d0986b591abaea2d257f1797818b9387c79687f827d00aafd Nov 03 22:39:24.163964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.163905 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fca6c22-9dfb-49ec-8051-4dbd379742a8-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:24.163964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.163937 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzt9k\" (UniqueName: \"kubernetes.io/projected/2fca6c22-9dfb-49ec-8051-4dbd379742a8-kube-api-access-qzt9k\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:24.733206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.733170 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" event={"ID":"f55e5d3f-2802-49a3-a784-602e3d1c1327","Type":"ContainerStarted","Data":"a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1"} Nov 03 22:39:24.733757 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.733231 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" event={"ID":"f55e5d3f-2802-49a3-a784-602e3d1c1327","Type":"ContainerStarted","Data":"ee98209422826f9d0986b591abaea2d257f1797818b9387c79687f827d00aafd"} Nov 03 22:39:24.733757 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.733262 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:24.735098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.735074 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" Nov 03 22:39:24.735222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.735072 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz" event={"ID":"2fca6c22-9dfb-49ec-8051-4dbd379742a8","Type":"ContainerDied","Data":"febdb58b15ac87eef593a8ac5aee7f9e3ab26f90704912627563757adfe1af1e"} Nov 03 22:39:24.735222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.735212 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="febdb58b15ac87eef593a8ac5aee7f9e3ab26f90704912627563757adfe1af1e" Nov 03 22:39:24.775819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:24.775770 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" podStartSLOduration=1.7757566919999999 podStartE2EDuration="1.775756692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:24.774686907 +0000 UTC m=+1591.178627882" watchObservedRunningTime="2026-11-03 22:39:24.775756692 +0000 UTC m=+1591.179697667" Nov 03 22:39:26.952607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:26.952572 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7"] Nov 03 22:39:26.953042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:26.952839 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" containerID="cri-o://83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e" gracePeriod=30 Nov 03 22:39:30.700781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:30.700743 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj"] Nov 03 22:39:30.738122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:30.738086 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6"] Nov 03 22:39:32.324800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.324763 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:39:32.434519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.434485 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-kube-api-access-jgwdg\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-kube-api-access-jgwdg\") pod \"f335df24-eab6-49f0-a307-86fea13f9402\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " Nov 03 22:39:32.434711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.434585 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-envoy\") pod \"f335df24-eab6-49f0-a307-86fea13f9402\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " Nov 03 22:39:32.434711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.434629 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-data\") pod \"f335df24-eab6-49f0-a307-86fea13f9402\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " Nov 03 22:39:32.434711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.434687 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-credential-socket\") pod \"f335df24-eab6-49f0-a307-86fea13f9402\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " Nov 03 22:39:32.434880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.434765 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-istio-token\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-istio-token\") pod \"f335df24-eab6-49f0-a307-86fea13f9402\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " Nov 03 22:39:32.434880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.434816 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-socket\") pod \"f335df24-eab6-49f0-a307-86fea13f9402\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " Nov 03 22:39:32.434880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.434861 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f335df24-eab6-49f0-a307-86fea13f9402-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f335df24-eab6-49f0-a307-86fea13f9402-istio-podinfo\") pod \"f335df24-eab6-49f0-a307-86fea13f9402\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " Nov 03 22:39:32.435028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.434915 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f335df24-eab6-49f0-a307-86fea13f9402-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f335df24-eab6-49f0-a307-86fea13f9402-istiod-ca-cert\") pod \"f335df24-eab6-49f0-a307-86fea13f9402\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " Nov 03 22:39:32.435028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.434958 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-certs\") pod \"f335df24-eab6-49f0-a307-86fea13f9402\" (UID: \"f335df24-eab6-49f0-a307-86fea13f9402\") " Nov 03 22:39:32.435469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.435438 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-data" pod "f335df24-eab6-49f0-a307-86fea13f9402" (UID: "f335df24-eab6-49f0-a307-86fea13f9402"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.435604 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.435522 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-certs" pod "f335df24-eab6-49f0-a307-86fea13f9402" (UID: "f335df24-eab6-49f0-a307-86fea13f9402"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.435604 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.435541 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-socket" pod "f335df24-eab6-49f0-a307-86fea13f9402" (UID: "f335df24-eab6-49f0-a307-86fea13f9402"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.435728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.435620 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-credential-socket" pod "f335df24-eab6-49f0-a307-86fea13f9402" (UID: "f335df24-eab6-49f0-a307-86fea13f9402"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.435854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.435836 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.435942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.435855 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.435942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.435867 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.435942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.435875 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.436258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.436238 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f335df24-eab6-49f0-a307-86fea13f9402-istiod-ca-cert" pod "f335df24-eab6-49f0-a307-86fea13f9402" (UID: "f335df24-eab6-49f0-a307-86fea13f9402"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:32.437228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.437206 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f335df24-eab6-49f0-a307-86fea13f9402-istio-podinfo" pod "f335df24-eab6-49f0-a307-86fea13f9402" (UID: "f335df24-eab6-49f0-a307-86fea13f9402"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:39:32.437490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.437468 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-kube-api-access-jgwdg" pod "f335df24-eab6-49f0-a307-86fea13f9402" (UID: "f335df24-eab6-49f0-a307-86fea13f9402"). InnerVolumeSpecName "kube-api-access-jgwdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:32.437666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.437636 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-envoy" pod "f335df24-eab6-49f0-a307-86fea13f9402" (UID: "f335df24-eab6-49f0-a307-86fea13f9402"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:39:32.438008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.437988 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-istio-token" pod "f335df24-eab6-49f0-a307-86fea13f9402" (UID: "f335df24-eab6-49f0-a307-86fea13f9402"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:32.537463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.537370 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f335df24-eab6-49f0-a307-86fea13f9402-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.537463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.537403 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.537463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.537415 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f335df24-eab6-49f0-a307-86fea13f9402-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.537463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.537430 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f335df24-eab6-49f0-a307-86fea13f9402-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.537463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.537443 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgwdg\" (UniqueName: \"kubernetes.io/projected/f335df24-eab6-49f0-a307-86fea13f9402-kube-api-access-jgwdg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:32.816394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.816307 4903 generic.go:356] "Generic (PLEG): container finished" podID="f335df24-eab6-49f0-a307-86fea13f9402" containerID="83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e" exitCode=0 Nov 03 22:39:32.816394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.816360 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" Nov 03 22:39:32.816394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.816366 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" event={"ID":"f335df24-eab6-49f0-a307-86fea13f9402","Type":"ContainerDied","Data":"83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e"} Nov 03 22:39:32.816621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.816408 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7" event={"ID":"f335df24-eab6-49f0-a307-86fea13f9402","Type":"ContainerDied","Data":"a59e0f7f21a0da5622538cf74d5923bb1f3cb495d315bfa1b40d6228c4b10caf"} Nov 03 22:39:32.816621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.816428 4903 scope.go:122] "RemoveContainer" containerID="83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e" Nov 03 22:39:32.829981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.829965 4903 scope.go:122] "RemoveContainer" containerID="58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a" Nov 03 22:39:32.840008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.839988 4903 scope.go:122] "RemoveContainer" containerID="83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e" Nov 03 22:39:32.840270 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:39:32.840253 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e\": container with ID starting with 83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e not found: ID does not exist" containerID="83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e" Nov 03 22:39:32.840391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.840278 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e"} err="failed to get container status \"83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e\": rpc error: code = NotFound desc = could not find container \"83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e\": container with ID starting with 83c9bd443baf3331bd75de35c9a656c6c7df87d1e97d9b7357ed7a1906ceab3e not found: ID does not exist" Nov 03 22:39:32.840391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.840320 4903 scope.go:122] "RemoveContainer" containerID="58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a" Nov 03 22:39:32.862331 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:39:32.860540 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a\": container with ID starting with 58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a not found: ID does not exist" containerID="58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a" Nov 03 22:39:32.862331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.860579 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a"} err="failed to get container status \"58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a\": rpc error: code = NotFound desc = could not find container \"58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a\": container with ID starting with 58f6cb3ed3ef6401472d1f1097aa8f1109604d9037792ab55d04d42799950c1a not found: ID does not exist" Nov 03 22:39:32.884308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.884273 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7"] Nov 03 22:39:32.894732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:32.894707 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8b1efeb6-openshift-default-5cf66d64d5-z8hk7"] Nov 03 22:39:33.782426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:33.782386 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f335df24-eab6-49f0-a307-86fea13f9402" path="/var/lib/kubelet/pods/f335df24-eab6-49f0-a307-86fea13f9402/volumes" Nov 03 22:39:35.701384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:35.701341 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" podUID="2889ddb8-cf2b-4d0a-bb3a-49447d395d2b" containerName="payload-processing" containerID="cri-o://22776e5f336c169d0a60dec5a0adfd4c3f64adba86d486b4576f2fa097e1949d" gracePeriod=25 Nov 03 22:39:35.738613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:35.738561 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" podUID="a4912538-fee8-4c7e-8b44-fff393346bf0" containerName="payload-pre-processing" containerID="cri-o://45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce" gracePeriod=25 Nov 03 22:39:35.760621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:35.759250 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:39:35.885801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:35.885769 4903 generic.go:356] "Generic (PLEG): container finished" podID="2889ddb8-cf2b-4d0a-bb3a-49447d395d2b" containerID="22776e5f336c169d0a60dec5a0adfd4c3f64adba86d486b4576f2fa097e1949d" exitCode=0 Nov 03 22:39:35.885923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:35.885882 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" event={"ID":"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b","Type":"ContainerDied","Data":"22776e5f336c169d0a60dec5a0adfd4c3f64adba86d486b4576f2fa097e1949d"} Nov 03 22:39:36.134798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.134754 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:39:36.208667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.208641 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:39:36.223239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.223178 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-kube-api-access-qzkcr\" (UniqueName: \"kubernetes.io/projected/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-kube-api-access-qzkcr\") pod \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\" (UID: \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\") " Nov 03 22:39:36.223239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.223214 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-plugins-config-volume\") pod \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\" (UID: \"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b\") " Nov 03 22:39:36.224404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.224372 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-plugins-config-volume" pod "2889ddb8-cf2b-4d0a-bb3a-49447d395d2b" (UID: "2889ddb8-cf2b-4d0a-bb3a-49447d395d2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:36.225938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.225914 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-kube-api-access-qzkcr" pod "2889ddb8-cf2b-4d0a-bb3a-49447d395d2b" (UID: "2889ddb8-cf2b-4d0a-bb3a-49447d395d2b"). InnerVolumeSpecName "kube-api-access-qzkcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:36.325349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.325285 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4912538-fee8-4c7e-8b44-fff393346bf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4912538-fee8-4c7e-8b44-fff393346bf0-plugins-config-volume\") pod \"a4912538-fee8-4c7e-8b44-fff393346bf0\" (UID: \"a4912538-fee8-4c7e-8b44-fff393346bf0\") " Nov 03 22:39:36.325545 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.325449 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4912538-fee8-4c7e-8b44-fff393346bf0-kube-api-access-x5znr\" (UniqueName: \"kubernetes.io/projected/a4912538-fee8-4c7e-8b44-fff393346bf0-kube-api-access-x5znr\") pod \"a4912538-fee8-4c7e-8b44-fff393346bf0\" (UID: \"a4912538-fee8-4c7e-8b44-fff393346bf0\") " Nov 03 22:39:36.326008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.325981 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4912538-fee8-4c7e-8b44-fff393346bf0-plugins-config-volume" pod "a4912538-fee8-4c7e-8b44-fff393346bf0" (UID: "a4912538-fee8-4c7e-8b44-fff393346bf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:39:36.326334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.326316 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzkcr\" (UniqueName: \"kubernetes.io/projected/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-kube-api-access-qzkcr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:36.326457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.326448 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:36.326511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.326504 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4912538-fee8-4c7e-8b44-fff393346bf0-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:36.327804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.327779 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4912538-fee8-4c7e-8b44-fff393346bf0-kube-api-access-x5znr" pod "a4912538-fee8-4c7e-8b44-fff393346bf0" (UID: "a4912538-fee8-4c7e-8b44-fff393346bf0"). InnerVolumeSpecName "kube-api-access-x5znr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:39:36.427821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.427790 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5znr\" (UniqueName: \"kubernetes.io/projected/a4912538-fee8-4c7e-8b44-fff393346bf0-kube-api-access-x5znr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:39:36.896675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.896643 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" Nov 03 22:39:36.897132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.896684 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj" event={"ID":"2889ddb8-cf2b-4d0a-bb3a-49447d395d2b","Type":"ContainerDied","Data":"bf2740208e2816b8757581cfe6c7ea768300b3b965bc3db898ade74953273a1e"} Nov 03 22:39:36.897132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.896756 4903 scope.go:122] "RemoveContainer" containerID="22776e5f336c169d0a60dec5a0adfd4c3f64adba86d486b4576f2fa097e1949d" Nov 03 22:39:36.900312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.898946 4903 generic.go:356] "Generic (PLEG): container finished" podID="a4912538-fee8-4c7e-8b44-fff393346bf0" containerID="45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce" exitCode=0 Nov 03 22:39:36.900312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.898983 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" Nov 03 22:39:36.900312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.899020 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" event={"ID":"a4912538-fee8-4c7e-8b44-fff393346bf0","Type":"ContainerDied","Data":"45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce"} Nov 03 22:39:36.900312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.899066 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6" event={"ID":"a4912538-fee8-4c7e-8b44-fff393346bf0","Type":"ContainerDied","Data":"314ff03a94babf0c6630f74c4271f697b695891c513ed2d504a6615aa8002d59"} Nov 03 22:39:36.913433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.911788 4903 scope.go:122] "RemoveContainer" containerID="45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce" Nov 03 22:39:36.924707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.924687 4903 scope.go:122] "RemoveContainer" containerID="45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce" Nov 03 22:39:36.925004 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:39:36.924983 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce\": container with ID starting with 45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce not found: ID does not exist" containerID="45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce" Nov 03 22:39:36.925059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.925012 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce"} err="failed to get container status \"45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce\": rpc error: code = NotFound desc = could not find container \"45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce\": container with ID starting with 45af545a3cf1ca4fe770fade77bdda0ad71d35b9b730fa08f4e62e2d58dcc0ce not found: ID does not exist" Nov 03 22:39:36.981224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.981129 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj"] Nov 03 22:39:36.984389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:36.984359 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5dd3b1-5b45c4ff8-rmshj"] Nov 03 22:39:37.005487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:37.005456 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6"] Nov 03 22:39:37.010383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:37.010359 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-5dd3b1-7bdc56b4f5-cxmt6"] Nov 03 22:39:37.739202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:37.739169 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2889ddb8-cf2b-4d0a-bb3a-49447d395d2b" path="/var/lib/kubelet/pods/2889ddb8-cf2b-4d0a-bb3a-49447d395d2b/volumes" Nov 03 22:39:37.739903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:37.739884 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4912538-fee8-4c7e-8b44-fff393346bf0" path="/var/lib/kubelet/pods/a4912538-fee8-4c7e-8b44-fff393346bf0/volumes" Nov 03 22:39:41.039110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:41.033201 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll"] Nov 03 22:39:41.060877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:41.042808 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baeaa733-ldbll"] Nov 03 22:39:41.736799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:41.736757 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2657aefc-f8ec-4dfa-8762-bfbc6322542c" path="/var/lib/kubelet/pods/2657aefc-f8ec-4dfa-8762-bfbc6322542c/volumes" Nov 03 22:39:41.949737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:41.949704 4903 generic.go:356] "Generic (PLEG): container finished" podID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerID="a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8" exitCode=0 Nov 03 22:39:41.949923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:41.949750 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" event={"ID":"bbae0ea0-7a75-4080-96b9-aca308be40ec","Type":"ContainerDied","Data":"a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8"} Nov 03 22:39:41.950194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:41.950181 4903 scope.go:122] "RemoveContainer" containerID="a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8" Nov 03 22:39:41.952146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:41.952111 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:41.952261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:41.952197 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:41.952261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:41.952233 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:39:42.978949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:42.978849 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" event={"ID":"bbae0ea0-7a75-4080-96b9-aca308be40ec","Type":"ContainerStarted","Data":"13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064"} Nov 03 22:39:43.586978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:43.586942 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:39:43.588003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:43.587974 4903 patch_prober.go:29] interesting pod/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:39:43.588164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:43.588027 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 22:39:44.586803 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:44.586766 4903 patch_prober.go:29] interesting pod/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:39:44.587236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:44.586823 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 22:39:45.586991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:45.586941 4903 patch_prober.go:29] interesting pod/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:39:45.587429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:45.587022 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 22:39:46.587052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:46.586972 4903 patch_prober.go:29] interesting pod/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:39:46.587052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:46.587034 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 22:39:47.590783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:47.590737 4903 patch_prober.go:29] interesting pod/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Nov 03 22:39:47.591401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:47.590826 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Nov 03 22:39:48.298162 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:48.298128 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:39:48.386117 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:39:48.386078 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:39:48.592238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:48.592164 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:39:49.029410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:49.029385 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:39:49.621556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:49.621518 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm"] Nov 03 22:39:49.626209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:49.626182 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c0223-jbggm"] Nov 03 22:39:49.752020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:49.751984 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aba7049-e353-4a86-8699-c8cf726b4fc8" path="/var/lib/kubelet/pods/3aba7049-e353-4a86-8699-c8cf726b4fc8/volumes" Nov 03 22:39:53.146038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146008 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k"] Nov 03 22:39:53.146880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146861 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" Nov 03 22:39:53.146880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146879 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" Nov 03 22:39:53.147024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146896 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" Nov 03 22:39:53.147024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146901 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" Nov 03 22:39:53.147024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146930 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2889ddb8-cf2b-4d0a-bb3a-49447d395d2b" containerName="payload-processing" Nov 03 22:39:53.147024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146935 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2889ddb8-cf2b-4d0a-bb3a-49447d395d2b" containerName="payload-processing" Nov 03 22:39:53.147024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146945 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a4912538-fee8-4c7e-8b44-fff393346bf0" containerName="payload-pre-processing" Nov 03 22:39:53.147024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146951 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4912538-fee8-4c7e-8b44-fff393346bf0" containerName="payload-pre-processing" Nov 03 22:39:53.147024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146968 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2fca6c22-9dfb-49ec-8051-4dbd379742a8" containerName="revoke-keys" Nov 03 22:39:53.147024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.146973 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fca6c22-9dfb-49ec-8051-4dbd379742a8" containerName="revoke-keys" Nov 03 22:39:53.147266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.147090 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2889ddb8-cf2b-4d0a-bb3a-49447d395d2b" containerName="payload-processing" Nov 03 22:39:53.147266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.147099 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4912538-fee8-4c7e-8b44-fff393346bf0" containerName="payload-pre-processing" Nov 03 22:39:53.147266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.147109 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fca6c22-9dfb-49ec-8051-4dbd379742a8" containerName="revoke-keys" Nov 03 22:39:53.147266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.147118 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" Nov 03 22:39:53.147266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.147125 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f335df24-eab6-49f0-a307-86fea13f9402" containerName="istio-proxy" Nov 03 22:39:53.150822 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.150805 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:39:53.165576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.165554 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k"] Nov 03 22:39:53.186894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.186867 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0a7065\"" Nov 03 22:39:53.270696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.270661 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0a7065-dockercfg-nfcqf\"" Nov 03 22:39:53.392896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.392853 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642bd136-2764-4607-9048-d9103f23d8a6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k\" (UID: \"642bd136-2764-4607-9048-d9103f23d8a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:39:53.393110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.393089 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9528b\" (UniqueName: \"kubernetes.io/projected/642bd136-2764-4607-9048-d9103f23d8a6-kube-api-access-9528b\") pod \"payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k\" (UID: \"642bd136-2764-4607-9048-d9103f23d8a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:39:53.495224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.495185 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642bd136-2764-4607-9048-d9103f23d8a6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k\" (UID: \"642bd136-2764-4607-9048-d9103f23d8a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:39:53.495461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.495331 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9528b\" (UniqueName: \"kubernetes.io/projected/642bd136-2764-4607-9048-d9103f23d8a6-kube-api-access-9528b\") pod \"payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k\" (UID: \"642bd136-2764-4607-9048-d9103f23d8a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:39:53.495994 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.495973 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642bd136-2764-4607-9048-d9103f23d8a6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k\" (UID: \"642bd136-2764-4607-9048-d9103f23d8a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:39:53.517539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.517495 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9528b\" (UniqueName: \"kubernetes.io/projected/642bd136-2764-4607-9048-d9103f23d8a6-kube-api-access-9528b\") pod \"payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k\" (UID: \"642bd136-2764-4607-9048-d9103f23d8a6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:39:53.777355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.777267 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0a7065-dockercfg-nfcqf\"" Nov 03 22:39:53.786546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:53.786516 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:39:54.102491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:54.102457 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k"] Nov 03 22:39:55.086458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:55.086418 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" event={"ID":"642bd136-2764-4607-9048-d9103f23d8a6","Type":"ContainerStarted","Data":"ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438"} Nov 03 22:39:55.086876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:55.086483 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" event={"ID":"642bd136-2764-4607-9048-d9103f23d8a6","Type":"ContainerStarted","Data":"c3d681d1a8c64fadf9fa526f13b862d5627219ee0dd8cd596e234b31138f7ca5"} Nov 03 22:39:55.086876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:55.086510 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:39:58.687904 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.687844 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" podStartSLOduration=5.687829147 podStartE2EDuration="5.687829147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:39:55.123995554 +0000 UTC m=+1621.527936532" watchObservedRunningTime="2026-11-03 22:39:58.687829147 +0000 UTC m=+1625.091770156" Nov 03 22:39:58.690373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.690347 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b"] Nov 03 22:39:58.695689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.695669 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:58.709151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.709121 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9pqw\" (UniqueName: \"kubernetes.io/projected/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-kube-api-access-p9pqw\") pod \"maas-api-revoke-keys-e2e-api-b-22edc1-54q9b\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:58.709474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.709448 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22edc1-54q9b\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:58.709585 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.709488 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22edc1-54q9b\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:58.726915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.726893 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:39:58.727061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.726926 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:39:58.734427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.734386 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b"] Nov 03 22:39:58.735979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.735951 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:39:58.736114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.736050 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:39:58.811438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.811403 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22edc1-54q9b\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:58.811623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.811454 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22edc1-54q9b\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:58.811623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.811517 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pqw\" (UniqueName: \"kubernetes.io/projected/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-kube-api-access-p9pqw\") pod \"maas-api-revoke-keys-e2e-api-b-22edc1-54q9b\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:58.812073 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.812053 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22edc1-54q9b\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:58.813754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.813733 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-22edc1-54q9b\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:58.833018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:58.832989 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pqw\" (UniqueName: \"kubernetes.io/projected/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-kube-api-access-p9pqw\") pod \"maas-api-revoke-keys-e2e-api-b-22edc1-54q9b\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:59.010480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:59.010391 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:39:59.507343 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:39:59.507315 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb58ca4d_c7e8_49a2_92ce_5ebff00ab93e.slice/crio-68462dc446df7a0be23e94baa619555588961a5ad27fbc430f76bac48c2b7fbd WatchSource:0}: Error finding container 68462dc446df7a0be23e94baa619555588961a5ad27fbc430f76bac48c2b7fbd: Status 404 returned error can't find the container with id 68462dc446df7a0be23e94baa619555588961a5ad27fbc430f76bac48c2b7fbd Nov 03 22:39:59.508980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:39:59.508958 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b"] Nov 03 22:40:00.186197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:00.186158 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" event={"ID":"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e","Type":"ContainerStarted","Data":"75948a0ec88ca19a7595ee9194708583006378b41a0e9d835a863b18f487960e"} Nov 03 22:40:00.186656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:00.186259 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" event={"ID":"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e","Type":"ContainerStarted","Data":"68462dc446df7a0be23e94baa619555588961a5ad27fbc430f76bac48c2b7fbd"} Nov 03 22:40:00.226208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:00.226154 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" podStartSLOduration=2.226141724 podStartE2EDuration="2.226141724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:00.222983737 +0000 UTC m=+1626.626924713" watchObservedRunningTime="2026-11-03 22:40:00.226141724 +0000 UTC m=+1626.630082731" Nov 03 22:40:01.197268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:01.197236 4903 generic.go:356] "Generic (PLEG): container finished" podID="bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e" containerID="75948a0ec88ca19a7595ee9194708583006378b41a0e9d835a863b18f487960e" exitCode=0 Nov 03 22:40:01.197722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:01.197320 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" event={"ID":"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e","Type":"ContainerDied","Data":"75948a0ec88ca19a7595ee9194708583006378b41a0e9d835a863b18f487960e"} Nov 03 22:40:02.377489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.377455 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:40:02.512670 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.512587 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-openshift-ca\") pod \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " Nov 03 22:40:02.512836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.512764 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-kube-api-access-p9pqw\" (UniqueName: \"kubernetes.io/projected/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-kube-api-access-p9pqw\") pod \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " Nov 03 22:40:02.512892 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.512847 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-xks-ca\") pod \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\" (UID: \"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e\") " Nov 03 22:40:02.512961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.512934 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-openshift-ca" pod "bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e" (UID: "bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:02.513548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.513529 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:02.515641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.515613 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-kube-api-access-p9pqw" pod "bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e" (UID: "bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e"). InnerVolumeSpecName "kube-api-access-p9pqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:02.515749 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.515655 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-xks-ca" pod "bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e" (UID: "bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:40:02.614819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.614780 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9pqw\" (UniqueName: \"kubernetes.io/projected/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-kube-api-access-p9pqw\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:02.614819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:02.614815 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:03.217696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:03.217666 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" Nov 03 22:40:03.217889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:03.217700 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b" event={"ID":"bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e","Type":"ContainerDied","Data":"68462dc446df7a0be23e94baa619555588961a5ad27fbc430f76bac48c2b7fbd"} Nov 03 22:40:03.217889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:03.217750 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68462dc446df7a0be23e94baa619555588961a5ad27fbc430f76bac48c2b7fbd" Nov 03 22:40:06.098490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:06.098453 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:40:09.433926 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:09.433877 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2"] Nov 03 22:40:09.478135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:09.478093 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47"] Nov 03 22:40:12.010439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.010387 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Nov 03 22:40:12.053840 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.050392 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.121464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.121410 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.122046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.122014 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 03 22:40:12.160508 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.160465 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 03 22:40:12.196428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.196383 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.199448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.199406 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" podUID="40bf741a-dbda-425a-94b3-27e46a7d1a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.205451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.205418 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.288378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.287435 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.454316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.454188 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.456023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.455981 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.514449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.514405 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.573488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.573386 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.589470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.589428 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": context deadline exceeded" Nov 03 22:40:12.630693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.630651 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.688659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.687407 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" podUID="10347389-d227-4891-9a5c-401dec199b97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.207:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.736315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.734363 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:12.736315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.734679 4903 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": context deadline exceeded" Nov 03 22:40:12.777319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:12.756221 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 03 22:40:13.218500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:13.218451 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:40:13.219865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:13.219835 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:40:13.302821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:13.302787 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 22:40:14.439017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.438978 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" podUID="f81d958b-c19c-4f8c-97f4-4d6a5a46a921" containerName="payload-processing" containerID="cri-o://6157ce885155799b59e483136347e9e0c510201968033d058f032462b05dfe5d" gracePeriod=25 Nov 03 22:40:14.479165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.479091 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" podUID="07596a42-3f87-4a8c-9362-d78b4cb27e62" containerName="payload-pre-processing" containerID="cri-o://b707aa2f2ef8ffe1111479101871f1f459623d47cdef5b31e04eb4e2d60b2e10" gracePeriod=25 Nov 03 22:40:14.720587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.720280 4903 generic.go:356] "Generic (PLEG): container finished" podID="07596a42-3f87-4a8c-9362-d78b4cb27e62" containerID="b707aa2f2ef8ffe1111479101871f1f459623d47cdef5b31e04eb4e2d60b2e10" exitCode=0 Nov 03 22:40:14.720587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.720452 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" event={"ID":"07596a42-3f87-4a8c-9362-d78b4cb27e62","Type":"ContainerDied","Data":"b707aa2f2ef8ffe1111479101871f1f459623d47cdef5b31e04eb4e2d60b2e10"} Nov 03 22:40:14.723572 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.723354 4903 generic.go:356] "Generic (PLEG): container finished" podID="f81d958b-c19c-4f8c-97f4-4d6a5a46a921" containerID="6157ce885155799b59e483136347e9e0c510201968033d058f032462b05dfe5d" exitCode=0 Nov 03 22:40:14.723572 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.723487 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" event={"ID":"f81d958b-c19c-4f8c-97f4-4d6a5a46a921","Type":"ContainerDied","Data":"6157ce885155799b59e483136347e9e0c510201968033d058f032462b05dfe5d"} Nov 03 22:40:14.853218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.852828 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:40:14.939049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.939010 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-plugins-config-volume\") pod \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\" (UID: \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\") " Nov 03 22:40:14.939260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.939241 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-kube-api-access-f827w\" (UniqueName: \"kubernetes.io/projected/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-kube-api-access-f827w\") pod \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\" (UID: \"f81d958b-c19c-4f8c-97f4-4d6a5a46a921\") " Nov 03 22:40:14.940669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.940637 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-plugins-config-volume" pod "f81d958b-c19c-4f8c-97f4-4d6a5a46a921" (UID: "f81d958b-c19c-4f8c-97f4-4d6a5a46a921"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:14.942730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.942704 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-kube-api-access-f827w" pod "f81d958b-c19c-4f8c-97f4-4d6a5a46a921" (UID: "f81d958b-c19c-4f8c-97f4-4d6a5a46a921"). InnerVolumeSpecName "kube-api-access-f827w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:14.950418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:14.950398 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:40:15.041021 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.040987 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07596a42-3f87-4a8c-9362-d78b4cb27e62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07596a42-3f87-4a8c-9362-d78b4cb27e62-plugins-config-volume\") pod \"07596a42-3f87-4a8c-9362-d78b4cb27e62\" (UID: \"07596a42-3f87-4a8c-9362-d78b4cb27e62\") " Nov 03 22:40:15.041220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.041133 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07596a42-3f87-4a8c-9362-d78b4cb27e62-kube-api-access-l5rvr\" (UniqueName: \"kubernetes.io/projected/07596a42-3f87-4a8c-9362-d78b4cb27e62-kube-api-access-l5rvr\") pod \"07596a42-3f87-4a8c-9362-d78b4cb27e62\" (UID: \"07596a42-3f87-4a8c-9362-d78b4cb27e62\") " Nov 03 22:40:15.041473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.041447 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07596a42-3f87-4a8c-9362-d78b4cb27e62-plugins-config-volume" pod "07596a42-3f87-4a8c-9362-d78b4cb27e62" (UID: "07596a42-3f87-4a8c-9362-d78b4cb27e62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:15.042790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.042760 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f827w\" (UniqueName: \"kubernetes.io/projected/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-kube-api-access-f827w\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:15.042911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.042795 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07596a42-3f87-4a8c-9362-d78b4cb27e62-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:15.042911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.042812 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81d958b-c19c-4f8c-97f4-4d6a5a46a921-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:15.043830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.043800 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07596a42-3f87-4a8c-9362-d78b4cb27e62-kube-api-access-l5rvr" pod "07596a42-3f87-4a8c-9362-d78b4cb27e62" (UID: "07596a42-3f87-4a8c-9362-d78b4cb27e62"). InnerVolumeSpecName "kube-api-access-l5rvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:15.145005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.144968 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5rvr\" (UniqueName: \"kubernetes.io/projected/07596a42-3f87-4a8c-9362-d78b4cb27e62-kube-api-access-l5rvr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:15.735893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.735810 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" Nov 03 22:40:15.737855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.737822 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" Nov 03 22:40:15.742322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.742197 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2" event={"ID":"f81d958b-c19c-4f8c-97f4-4d6a5a46a921","Type":"ContainerDied","Data":"59ff0eec42110b93649b42950dc12a751da00a929647080abd3a769d64a9f84e"} Nov 03 22:40:15.742322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.742255 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47" event={"ID":"07596a42-3f87-4a8c-9362-d78b4cb27e62","Type":"ContainerDied","Data":"4181afd0870c59584b3472d9e5faea641423af6a7ffeb5ac725b014cd4cd3ee1"} Nov 03 22:40:15.742322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.742311 4903 scope.go:122] "RemoveContainer" containerID="6157ce885155799b59e483136347e9e0c510201968033d058f032462b05dfe5d" Nov 03 22:40:15.772047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.772016 4903 scope.go:122] "RemoveContainer" containerID="b707aa2f2ef8ffe1111479101871f1f459623d47cdef5b31e04eb4e2d60b2e10" Nov 03 22:40:15.800940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.800912 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2"] Nov 03 22:40:15.813116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.813082 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-22edc1-998976c8b-4whv2"] Nov 03 22:40:15.846195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.846157 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47"] Nov 03 22:40:15.857308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:15.857253 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-22edc1-5bb4668bc-59h47"] Nov 03 22:40:17.739521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:17.739484 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07596a42-3f87-4a8c-9362-d78b4cb27e62" path="/var/lib/kubelet/pods/07596a42-3f87-4a8c-9362-d78b4cb27e62/volumes" Nov 03 22:40:17.740193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:17.740172 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81d958b-c19c-4f8c-97f4-4d6a5a46a921" path="/var/lib/kubelet/pods/f81d958b-c19c-4f8c-97f4-4d6a5a46a921/volumes" Nov 03 22:40:18.438976 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:40:18.438929 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:40:18.855665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:18.855557 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj"] Nov 03 22:40:18.867344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:18.867274 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b826b6-bcvtj"] Nov 03 22:40:19.736415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:19.736379 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="810ad376-f536-45ee-9624-1706cca2f59a" path="/var/lib/kubelet/pods/810ad376-f536-45ee-9624-1706cca2f59a/volumes" Nov 03 22:40:19.785244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:19.785214 4903 generic.go:356] "Generic (PLEG): container finished" podID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerID="349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841" exitCode=0 Nov 03 22:40:19.785460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:19.785323 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" event={"ID":"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e","Type":"ContainerDied","Data":"349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841"} Nov 03 22:40:19.785841 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:19.785825 4903 scope.go:122] "RemoveContainer" containerID="349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841" Nov 03 22:40:19.787744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:19.787714 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:40:19.787865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:19.787797 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:40:19.787865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:19.787853 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:40:20.796575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:20.796544 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" event={"ID":"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e","Type":"ContainerStarted","Data":"6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9"} Nov 03 22:40:20.866325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:20.865672 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:40:20.867270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:20.867234 4903 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 22:40:20.867472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:20.867355 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 22:40:21.857318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:21.856663 4903 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 22:40:21.857318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:21.856728 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 22:40:22.869700 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:22.869662 4903 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 22:40:22.870105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:22.869744 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 22:40:22.934836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:22.934802 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:40:23.857789 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:23.857756 4903 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 22:40:23.858075 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:23.858048 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 22:40:24.861207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:24.861169 4903 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 22:40:24.865447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:24.865405 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 22:40:25.853937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:25.853893 4903 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 22:40:25.854270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:25.854217 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 22:40:26.858389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:26.858359 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:40:26.859548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:26.859524 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:40:34.861081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.861043 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5"] Nov 03 22:40:34.861907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.861890 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f81d958b-c19c-4f8c-97f4-4d6a5a46a921" containerName="payload-processing" Nov 03 22:40:34.861963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.861908 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81d958b-c19c-4f8c-97f4-4d6a5a46a921" containerName="payload-processing" Nov 03 22:40:34.861963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.861924 4903 cpu_manager.go:414] "containerMap: removing container" podUID="07596a42-3f87-4a8c-9362-d78b4cb27e62" containerName="payload-pre-processing" Nov 03 22:40:34.861963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.861930 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07596a42-3f87-4a8c-9362-d78b4cb27e62" containerName="payload-pre-processing" Nov 03 22:40:34.861963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.861944 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e" containerName="revoke-keys" Nov 03 22:40:34.861963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.861949 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e" containerName="revoke-keys" Nov 03 22:40:34.862140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.862083 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81d958b-c19c-4f8c-97f4-4d6a5a46a921" containerName="payload-processing" Nov 03 22:40:34.862140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.862091 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="07596a42-3f87-4a8c-9362-d78b4cb27e62" containerName="payload-pre-processing" Nov 03 22:40:34.862140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.862098 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e" containerName="revoke-keys" Nov 03 22:40:34.865447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.865414 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:34.884823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.884796 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5"] Nov 03 22:40:34.933417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.933382 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7940ea-dockercfg-4z8pj\"" Nov 03 22:40:34.936210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.936182 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7940ea\"" Nov 03 22:40:34.964427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.964385 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b704aa-4636-40ca-819e-d7a18e55380a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5\" (UID: \"b8b704aa-4636-40ca-819e-d7a18e55380a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:34.965018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:34.964990 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rz2\" (UniqueName: \"kubernetes.io/projected/b8b704aa-4636-40ca-819e-d7a18e55380a-kube-api-access-55rz2\") pod \"payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5\" (UID: \"b8b704aa-4636-40ca-819e-d7a18e55380a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:35.070195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:35.067467 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b704aa-4636-40ca-819e-d7a18e55380a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5\" (UID: \"b8b704aa-4636-40ca-819e-d7a18e55380a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:35.070195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:35.067729 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rz2\" (UniqueName: \"kubernetes.io/projected/b8b704aa-4636-40ca-819e-d7a18e55380a-kube-api-access-55rz2\") pod \"payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5\" (UID: \"b8b704aa-4636-40ca-819e-d7a18e55380a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:35.070195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:35.069054 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b704aa-4636-40ca-819e-d7a18e55380a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5\" (UID: \"b8b704aa-4636-40ca-819e-d7a18e55380a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:35.128064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:35.127880 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rz2\" (UniqueName: \"kubernetes.io/projected/b8b704aa-4636-40ca-819e-d7a18e55380a-kube-api-access-55rz2\") pod \"payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5\" (UID: \"b8b704aa-4636-40ca-819e-d7a18e55380a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:35.186765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:35.186732 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:35.508573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:35.508540 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5"] Nov 03 22:40:35.509194 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:40:35.509050 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b704aa_4636_40ca_819e_d7a18e55380a.slice/crio-d06f4fe06dc6d4a51f12f2dfb56b1f2a330d2cabb66bd147639efc5eccba4818 WatchSource:0}: Error finding container d06f4fe06dc6d4a51f12f2dfb56b1f2a330d2cabb66bd147639efc5eccba4818: Status 404 returned error can't find the container with id d06f4fe06dc6d4a51f12f2dfb56b1f2a330d2cabb66bd147639efc5eccba4818 Nov 03 22:40:35.975317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:35.975171 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" event={"ID":"b8b704aa-4636-40ca-819e-d7a18e55380a","Type":"ContainerStarted","Data":"f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88"} Nov 03 22:40:35.975317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:35.975263 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" event={"ID":"b8b704aa-4636-40ca-819e-d7a18e55380a","Type":"ContainerStarted","Data":"d06f4fe06dc6d4a51f12f2dfb56b1f2a330d2cabb66bd147639efc5eccba4818"} Nov 03 22:40:35.975839 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:35.975556 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:36.013968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:36.013850 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" podStartSLOduration=2.013831597 podStartE2EDuration="2.013831597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:36.011999466 +0000 UTC m=+1662.415940441" watchObservedRunningTime="2026-11-03 22:40:36.013831597 +0000 UTC m=+1662.417772573" Nov 03 22:40:36.886610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:36.886576 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz"] Nov 03 22:40:36.894032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:36.894003 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5dd3b1-bgpsz"] Nov 03 22:40:37.740552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:37.740515 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fca6c22-9dfb-49ec-8051-4dbd379742a8" path="/var/lib/kubelet/pods/2fca6c22-9dfb-49ec-8051-4dbd379742a8/volumes" Nov 03 22:40:45.169026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:45.168989 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l"] Nov 03 22:40:45.169584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:45.169263 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" containerID="cri-o://13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064" gracePeriod=30 Nov 03 22:40:46.988198 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:46.988169 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:40:48.385678 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:40:48.385641 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:40:49.030730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:49.030697 4903 patch_prober.go:29] interesting pod/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:40:49.030905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:49.030768 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:40:50.479045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.479019 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:40:50.519864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.519835 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-certs\") pod \"bbae0ea0-7a75-4080-96b9-aca308be40ec\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " Nov 03 22:40:50.520033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.519960 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-kube-api-access-5ctg9\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-kube-api-access-5ctg9\") pod \"bbae0ea0-7a75-4080-96b9-aca308be40ec\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " Nov 03 22:40:50.520033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520002 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbae0ea0-7a75-4080-96b9-aca308be40ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbae0ea0-7a75-4080-96b9-aca308be40ec-istiod-ca-cert\") pod \"bbae0ea0-7a75-4080-96b9-aca308be40ec\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " Nov 03 22:40:50.520115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520103 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-token\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-token\") pod \"bbae0ea0-7a75-4080-96b9-aca308be40ec\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " Nov 03 22:40:50.520163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520113 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-certs" pod "bbae0ea0-7a75-4080-96b9-aca308be40ec" (UID: "bbae0ea0-7a75-4080-96b9-aca308be40ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:50.520163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520144 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-envoy\") pod \"bbae0ea0-7a75-4080-96b9-aca308be40ec\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " Nov 03 22:40:50.520253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520188 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-socket\") pod \"bbae0ea0-7a75-4080-96b9-aca308be40ec\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " Nov 03 22:40:50.520253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520228 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-credential-socket\") pod \"bbae0ea0-7a75-4080-96b9-aca308be40ec\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " Nov 03 22:40:50.520384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520254 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-data\") pod \"bbae0ea0-7a75-4080-96b9-aca308be40ec\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " Nov 03 22:40:50.520384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520361 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-podinfo\") pod \"bbae0ea0-7a75-4080-96b9-aca308be40ec\" (UID: \"bbae0ea0-7a75-4080-96b9-aca308be40ec\") " Nov 03 22:40:50.520565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520510 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-socket" pod "bbae0ea0-7a75-4080-96b9-aca308be40ec" (UID: "bbae0ea0-7a75-4080-96b9-aca308be40ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:50.520565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520540 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbae0ea0-7a75-4080-96b9-aca308be40ec-istiod-ca-cert" pod "bbae0ea0-7a75-4080-96b9-aca308be40ec" (UID: "bbae0ea0-7a75-4080-96b9-aca308be40ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:40:50.520758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520569 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-data" pod "bbae0ea0-7a75-4080-96b9-aca308be40ec" (UID: "bbae0ea0-7a75-4080-96b9-aca308be40ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:50.520758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.520587 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-credential-socket" pod "bbae0ea0-7a75-4080-96b9-aca308be40ec" (UID: "bbae0ea0-7a75-4080-96b9-aca308be40ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:50.521160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.521134 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:50.521160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.521164 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbae0ea0-7a75-4080-96b9-aca308be40ec-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:50.521389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.521177 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:50.521389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.521190 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:50.521389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.521203 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:50.522478 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.522449 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-envoy" pod "bbae0ea0-7a75-4080-96b9-aca308be40ec" (UID: "bbae0ea0-7a75-4080-96b9-aca308be40ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:40:50.522571 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.522478 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-token" pod "bbae0ea0-7a75-4080-96b9-aca308be40ec" (UID: "bbae0ea0-7a75-4080-96b9-aca308be40ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:50.522571 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.522514 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-kube-api-access-5ctg9" pod "bbae0ea0-7a75-4080-96b9-aca308be40ec" (UID: "bbae0ea0-7a75-4080-96b9-aca308be40ec"). InnerVolumeSpecName "kube-api-access-5ctg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:40:50.522758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.522735 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-podinfo" pod "bbae0ea0-7a75-4080-96b9-aca308be40ec" (UID: "bbae0ea0-7a75-4080-96b9-aca308be40ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:40:50.622741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.622706 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:50.622741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.622737 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ctg9\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-kube-api-access-5ctg9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:50.622741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.622747 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:50.622987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:50.622755 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbae0ea0-7a75-4080-96b9-aca308be40ec-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:40:51.046421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.046384 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99"] Nov 03 22:40:51.050638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.050609 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0cdebf2-k6s99"] Nov 03 22:40:51.148615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.148575 4903 generic.go:356] "Generic (PLEG): container finished" podID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerID="13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064" exitCode=0 Nov 03 22:40:51.148782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.148633 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" Nov 03 22:40:51.148782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.148652 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" event={"ID":"bbae0ea0-7a75-4080-96b9-aca308be40ec","Type":"ContainerDied","Data":"13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064"} Nov 03 22:40:51.148782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.148743 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l" event={"ID":"bbae0ea0-7a75-4080-96b9-aca308be40ec","Type":"ContainerDied","Data":"6c7cedc9b2787247566bebae00c4b4421a304fad51b6e36111fd11827a1b74a9"} Nov 03 22:40:51.148782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.148773 4903 scope.go:122] "RemoveContainer" containerID="13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064" Nov 03 22:40:51.161507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.161486 4903 scope.go:122] "RemoveContainer" containerID="a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8" Nov 03 22:40:51.172159 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.172135 4903 scope.go:122] "RemoveContainer" containerID="13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064" Nov 03 22:40:51.172504 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:40:51.172481 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064\": container with ID starting with 13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064 not found: ID does not exist" containerID="13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064" Nov 03 22:40:51.172552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.172515 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064"} err="failed to get container status \"13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064\": rpc error: code = NotFound desc = could not find container \"13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064\": container with ID starting with 13bbfe0ef6ec74b2c62a79a5a37b948471776b82ea4d6ecf2c07f0384cbb1064 not found: ID does not exist" Nov 03 22:40:51.172552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.172534 4903 scope.go:122] "RemoveContainer" containerID="a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8" Nov 03 22:40:51.172815 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:40:51.172793 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8\": container with ID starting with a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8 not found: ID does not exist" containerID="a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8" Nov 03 22:40:51.172899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.172820 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8"} err="failed to get container status \"a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8\": rpc error: code = NotFound desc = could not find container \"a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8\": container with ID starting with a972a9695b4e8f92d7d52a9c8c4ce85d8d5dcf33576fa885ce3889ce3c8d5be8 not found: ID does not exist" Nov 03 22:40:51.204706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.204679 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l"] Nov 03 22:40:51.214947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.214922 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5dd3b1-openshift-default-684dbd6c7c-k4v6l"] Nov 03 22:40:51.737778 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.737741 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" path="/var/lib/kubelet/pods/bbae0ea0-7a75-4080-96b9-aca308be40ec/volumes" Nov 03 22:40:51.738417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:51.738399 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5ca969-3ba7-47a7-b42c-b17c5a54f893" path="/var/lib/kubelet/pods/ee5ca969-3ba7-47a7-b42c-b17c5a54f893/volumes" Nov 03 22:40:52.270203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.270170 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth"] Nov 03 22:40:52.271495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.271467 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" Nov 03 22:40:52.271495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.271492 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" Nov 03 22:40:52.271793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.271776 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" Nov 03 22:40:52.271871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.271801 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" Nov 03 22:40:52.276667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.276642 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:40:52.296615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.296589 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth"] Nov 03 22:40:52.394385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.353596 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth\" (UID: \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:40:52.408318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.394846 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mz4x\" (UniqueName: \"kubernetes.io/projected/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-kube-api-access-8mz4x\") pod \"payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth\" (UID: \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:40:52.419323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.417463 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50b9b5\"" Nov 03 22:40:52.419323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.417626 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50b9b5-dockercfg-p4fzx\"" Nov 03 22:40:52.520367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.517318 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth\" (UID: \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:40:52.520367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.517492 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mz4x\" (UniqueName: \"kubernetes.io/projected/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-kube-api-access-8mz4x\") pod \"payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth\" (UID: \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:40:52.520367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.518934 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth\" (UID: \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:40:52.569193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.568671 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mz4x\" (UniqueName: \"kubernetes.io/projected/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-kube-api-access-8mz4x\") pod \"payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth\" (UID: \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:40:52.599002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.598969 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:40:52.916989 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:40:52.916939 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495cc6a9_b56d_41a2_82d4_3bb5998a84d1.slice/crio-324194690a485a5bece5a2aef1f50fcd8ac50a83f0ea45391e7dc4fd4731f7f6 WatchSource:0}: Error finding container 324194690a485a5bece5a2aef1f50fcd8ac50a83f0ea45391e7dc4fd4731f7f6: Status 404 returned error can't find the container with id 324194690a485a5bece5a2aef1f50fcd8ac50a83f0ea45391e7dc4fd4731f7f6 Nov 03 22:40:52.917675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:52.917650 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth"] Nov 03 22:40:53.167747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:53.167655 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" event={"ID":"495cc6a9-b56d-41a2-82d4-3bb5998a84d1","Type":"ContainerStarted","Data":"bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299"} Nov 03 22:40:53.167747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:53.167717 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" event={"ID":"495cc6a9-b56d-41a2-82d4-3bb5998a84d1","Type":"ContainerStarted","Data":"324194690a485a5bece5a2aef1f50fcd8ac50a83f0ea45391e7dc4fd4731f7f6"} Nov 03 22:40:53.167747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:53.167746 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:40:53.208562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:53.208513 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" podStartSLOduration=1.208498769 podStartE2EDuration="1.208498769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:40:53.204982962 +0000 UTC m=+1679.608923938" watchObservedRunningTime="2026-11-03 22:40:53.208498769 +0000 UTC m=+1679.612439741" Nov 03 22:40:57.923626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:40:57.923596 4903 scope.go:122] "RemoveContainer" containerID="16923e7522c0c398a59b619412c1f15ed50670b5ed762eb5dad1d49bafc12d45" Nov 03 22:41:03.158315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:03.158255 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:41:04.178409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:04.178377 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:41:09.484977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:09.484931 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b"] Nov 03 22:41:09.491239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:09.491208 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-22edc1-54q9b"] Nov 03 22:41:09.738176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:09.738099 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e" path="/var/lib/kubelet/pods/bb58ca4d-c7e8-49a2-92ce-5ebff00ab93e/volumes" Nov 03 22:41:11.395124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.395089 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5"] Nov 03 22:41:11.396423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.396396 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" Nov 03 22:41:11.396423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.396420 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbae0ea0-7a75-4080-96b9-aca308be40ec" containerName="istio-proxy" Nov 03 22:41:11.416338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.410017 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.418908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.418869 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5"] Nov 03 22:41:11.424990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.424953 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:41:11.435275 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.435243 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:41:11.435488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.435286 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:41:11.435488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.435354 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:41:11.497772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.497718 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0113354b-wp6f5\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.497772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.497773 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pqd\" (UniqueName: \"kubernetes.io/projected/1758bd64-657d-4cb3-a174-e28399527af7-kube-api-access-66pqd\") pod \"maas-api-revoke-sub-0113354b-wp6f5\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.498115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.497915 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0113354b-wp6f5\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.599825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.599787 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0113354b-wp6f5\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.599825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.599831 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pqd\" (UniqueName: \"kubernetes.io/projected/1758bd64-657d-4cb3-a174-e28399527af7-kube-api-access-66pqd\") pod \"maas-api-revoke-sub-0113354b-wp6f5\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.600071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.600045 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0113354b-wp6f5\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.600604 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.600585 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0113354b-wp6f5\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.602460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.602433 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0113354b-wp6f5\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.621907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.621879 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pqd\" (UniqueName: \"kubernetes.io/projected/1758bd64-657d-4cb3-a174-e28399527af7-kube-api-access-66pqd\") pod \"maas-api-revoke-sub-0113354b-wp6f5\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:11.754495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:11.754419 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:12.075971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:12.075936 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5"] Nov 03 22:41:12.076912 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:41:12.076883 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1758bd64_657d_4cb3_a174_e28399527af7.slice/crio-194cc99eb958002adad93cc9a88bbf09a5daa308f960caff9a846818226d6d6e WatchSource:0}: Error finding container 194cc99eb958002adad93cc9a88bbf09a5daa308f960caff9a846818226d6d6e: Status 404 returned error can't find the container with id 194cc99eb958002adad93cc9a88bbf09a5daa308f960caff9a846818226d6d6e Nov 03 22:41:12.447137 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:12.446663 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" event={"ID":"1758bd64-657d-4cb3-a174-e28399527af7","Type":"ContainerStarted","Data":"194cc99eb958002adad93cc9a88bbf09a5daa308f960caff9a846818226d6d6e"} Nov 03 22:41:13.517959 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:13.517920 4903 generic.go:356] "Generic (PLEG): container finished" podID="1758bd64-657d-4cb3-a174-e28399527af7" containerID="fe47d2caad43d4c3f6f691d072e8ebe607031bfc9a824216c99d8809fd26e2cc" exitCode=0 Nov 03 22:41:13.524953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:13.524826 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" event={"ID":"1758bd64-657d-4cb3-a174-e28399527af7","Type":"ContainerDied","Data":"fe47d2caad43d4c3f6f691d072e8ebe607031bfc9a824216c99d8809fd26e2cc"} Nov 03 22:41:14.749424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.749395 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:14.813781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.813554 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-openshift-ca" pod "1758bd64-657d-4cb3-a174-e28399527af7" (UID: "1758bd64-657d-4cb3-a174-e28399527af7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:41:14.817466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.817430 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-openshift-ca\") pod \"1758bd64-657d-4cb3-a174-e28399527af7\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " Nov 03 22:41:14.817728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.817559 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-xks-ca\") pod \"1758bd64-657d-4cb3-a174-e28399527af7\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " Nov 03 22:41:14.817728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.817645 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1758bd64-657d-4cb3-a174-e28399527af7-kube-api-access-66pqd\" (UniqueName: \"kubernetes.io/projected/1758bd64-657d-4cb3-a174-e28399527af7-kube-api-access-66pqd\") pod \"1758bd64-657d-4cb3-a174-e28399527af7\" (UID: \"1758bd64-657d-4cb3-a174-e28399527af7\") " Nov 03 22:41:14.820209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.819055 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:41:14.825173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.825132 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1758bd64-657d-4cb3-a174-e28399527af7-kube-api-access-66pqd" pod "1758bd64-657d-4cb3-a174-e28399527af7" (UID: "1758bd64-657d-4cb3-a174-e28399527af7"). InnerVolumeSpecName "kube-api-access-66pqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:41:14.831506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.831465 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-xks-ca" pod "1758bd64-657d-4cb3-a174-e28399527af7" (UID: "1758bd64-657d-4cb3-a174-e28399527af7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:41:14.921211 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.921172 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1758bd64-657d-4cb3-a174-e28399527af7-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:41:14.921211 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:14.921205 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66pqd\" (UniqueName: \"kubernetes.io/projected/1758bd64-657d-4cb3-a174-e28399527af7-kube-api-access-66pqd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:41:15.546378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:15.546341 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" event={"ID":"1758bd64-657d-4cb3-a174-e28399527af7","Type":"ContainerDied","Data":"194cc99eb958002adad93cc9a88bbf09a5daa308f960caff9a846818226d6d6e"} Nov 03 22:41:15.546378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:15.546372 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5" Nov 03 22:41:15.547688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:15.546403 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="194cc99eb958002adad93cc9a88bbf09a5daa308f960caff9a846818226d6d6e" Nov 03 22:41:16.556307 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:16.556263 4903 generic.go:356] "Generic (PLEG): container finished" podID="967c277a-6315-4353-8e39-b05adb456f1d" containerID="8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0" exitCode=0 Nov 03 22:41:16.556742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:16.556323 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" event={"ID":"967c277a-6315-4353-8e39-b05adb456f1d","Type":"ContainerDied","Data":"8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0"} Nov 03 22:41:16.556742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:16.556731 4903 scope.go:122] "RemoveContainer" containerID="8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0" Nov 03 22:41:16.558418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:16.558393 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:41:16.558488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:16.558460 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:41:16.558525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:16.558489 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:41:17.572869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:17.572827 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" event={"ID":"967c277a-6315-4353-8e39-b05adb456f1d","Type":"ContainerStarted","Data":"035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4"} Nov 03 22:41:18.200111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:18.200081 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:41:18.201592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:18.201568 4903 patch_prober.go:29] interesting pod/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:41:18.201700 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:18.201623 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 22:41:18.429096 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:41:18.428988 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:41:19.200912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:19.200880 4903 patch_prober.go:29] interesting pod/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:41:19.201359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:19.200938 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 22:41:19.589100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:19.589008 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:41:20.200271 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:20.200233 4903 patch_prober.go:29] interesting pod/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:41:20.200476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:20.200308 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 22:41:21.200926 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:21.200893 4903 patch_prober.go:29] interesting pod/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:41:21.201355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:21.200946 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 22:41:22.200610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:22.200575 4903 patch_prober.go:29] interesting pod/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:41:22.200931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:22.200883 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 22:41:23.199939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:23.199908 4903 patch_prober.go:29] interesting pod/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 22:41:23.200341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:23.199959 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 22:41:24.205606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:24.205576 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:41:24.206683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:24.206658 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:41:25.787098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.787060 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r"] Nov 03 22:41:25.787994 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.787977 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1758bd64-657d-4cb3-a174-e28399527af7" containerName="revoke-keys" Nov 03 22:41:25.788046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.787996 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1758bd64-657d-4cb3-a174-e28399527af7" containerName="revoke-keys" Nov 03 22:41:25.788164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.788154 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1758bd64-657d-4cb3-a174-e28399527af7" containerName="revoke-keys" Nov 03 22:41:25.791507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.791490 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:25.798782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.798758 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr"] Nov 03 22:41:25.802737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.802717 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:25.818769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.818745 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r"] Nov 03 22:41:25.821851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.821826 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bb1fa5-dockercfg-vs24q\"" Nov 03 22:41:25.821851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.821826 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bb1fa5\"" Nov 03 22:41:25.853166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.853131 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr"] Nov 03 22:41:25.912806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.912764 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqf7l\" (UniqueName: \"kubernetes.io/projected/7a0c63e2-19da-49d8-84b7-f826385c2d68-kube-api-access-sqf7l\") pod \"payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr\" (UID: \"7a0c63e2-19da-49d8-84b7-f826385c2d68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:25.917848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.913164 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea3c3cb-4508-409d-afb4-a03249c5702b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r\" (UID: \"cea3c3cb-4508-409d-afb4-a03249c5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:25.917848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.917268 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjctv\" (UniqueName: \"kubernetes.io/projected/cea3c3cb-4508-409d-afb4-a03249c5702b-kube-api-access-tjctv\") pod \"payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r\" (UID: \"cea3c3cb-4508-409d-afb4-a03249c5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:25.917848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:25.917516 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0c63e2-19da-49d8-84b7-f826385c2d68-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr\" (UID: \"7a0c63e2-19da-49d8-84b7-f826385c2d68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:26.029317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.020478 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqf7l\" (UniqueName: \"kubernetes.io/projected/7a0c63e2-19da-49d8-84b7-f826385c2d68-kube-api-access-sqf7l\") pod \"payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr\" (UID: \"7a0c63e2-19da-49d8-84b7-f826385c2d68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:26.029317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.020556 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea3c3cb-4508-409d-afb4-a03249c5702b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r\" (UID: \"cea3c3cb-4508-409d-afb4-a03249c5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:26.029317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.020710 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjctv\" (UniqueName: \"kubernetes.io/projected/cea3c3cb-4508-409d-afb4-a03249c5702b-kube-api-access-tjctv\") pod \"payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r\" (UID: \"cea3c3cb-4508-409d-afb4-a03249c5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:26.029317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.020870 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0c63e2-19da-49d8-84b7-f826385c2d68-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr\" (UID: \"7a0c63e2-19da-49d8-84b7-f826385c2d68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:26.029317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.021918 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0c63e2-19da-49d8-84b7-f826385c2d68-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr\" (UID: \"7a0c63e2-19da-49d8-84b7-f826385c2d68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:26.029317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.023021 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea3c3cb-4508-409d-afb4-a03249c5702b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r\" (UID: \"cea3c3cb-4508-409d-afb4-a03249c5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:26.065102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.065028 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjctv\" (UniqueName: \"kubernetes.io/projected/cea3c3cb-4508-409d-afb4-a03249c5702b-kube-api-access-tjctv\") pod \"payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r\" (UID: \"cea3c3cb-4508-409d-afb4-a03249c5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:26.073035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.073001 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqf7l\" (UniqueName: \"kubernetes.io/projected/7a0c63e2-19da-49d8-84b7-f826385c2d68-kube-api-access-sqf7l\") pod \"payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr\" (UID: \"7a0c63e2-19da-49d8-84b7-f826385c2d68\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:26.108161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.108126 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:26.119132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.119101 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:26.481873 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:41:26.481838 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea3c3cb_4508_409d_afb4_a03249c5702b.slice/crio-718fd7abdd81a191ac0d0cb66a943d580976ab9816c11f7621ec3a40f07c36af WatchSource:0}: Error finding container 718fd7abdd81a191ac0d0cb66a943d580976ab9816c11f7621ec3a40f07c36af: Status 404 returned error can't find the container with id 718fd7abdd81a191ac0d0cb66a943d580976ab9816c11f7621ec3a40f07c36af Nov 03 22:41:26.482184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.482158 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r"] Nov 03 22:41:26.496740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.496713 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr"] Nov 03 22:41:26.498921 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:41:26.498898 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0c63e2_19da_49d8_84b7_f826385c2d68.slice/crio-d10f83f23c4e01979d622502ebc99e8a71ae42601c84c90ba9000039f0733db9 WatchSource:0}: Error finding container d10f83f23c4e01979d622502ebc99e8a71ae42601c84c90ba9000039f0733db9: Status 404 returned error can't find the container with id d10f83f23c4e01979d622502ebc99e8a71ae42601c84c90ba9000039f0733db9 Nov 03 22:41:26.663941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.663902 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" event={"ID":"7a0c63e2-19da-49d8-84b7-f826385c2d68","Type":"ContainerStarted","Data":"39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73"} Nov 03 22:41:26.664140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.663970 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:26.664140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.663993 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" event={"ID":"7a0c63e2-19da-49d8-84b7-f826385c2d68","Type":"ContainerStarted","Data":"d10f83f23c4e01979d622502ebc99e8a71ae42601c84c90ba9000039f0733db9"} Nov 03 22:41:26.665924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.665900 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" event={"ID":"cea3c3cb-4508-409d-afb4-a03249c5702b","Type":"ContainerStarted","Data":"6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c"} Nov 03 22:41:26.666029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.665947 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" event={"ID":"cea3c3cb-4508-409d-afb4-a03249c5702b","Type":"ContainerStarted","Data":"718fd7abdd81a191ac0d0cb66a943d580976ab9816c11f7621ec3a40f07c36af"} Nov 03 22:41:26.666086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.666041 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:26.736946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.736462 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" podStartSLOduration=1.736443975 podStartE2EDuration="1.736443975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:26.732268146 +0000 UTC m=+1713.136209121" watchObservedRunningTime="2026-11-03 22:41:26.736443975 +0000 UTC m=+1713.140384952" Nov 03 22:41:26.814262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:26.814207 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" podStartSLOduration=1.8141904690000001 podStartE2EDuration="1.814190469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:26.800248261 +0000 UTC m=+1713.204189236" watchObservedRunningTime="2026-11-03 22:41:26.814190469 +0000 UTC m=+1713.218131444" Nov 03 22:41:37.685562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:37.685529 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:41:37.686124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:37.685601 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:41:47.593122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.593084 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4"] Nov 03 22:41:47.597143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.597119 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.612525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.612499 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0a7065\"/\"openshift-service-ca.crt\"" Nov 03 22:41:47.622237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.622208 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0a7065\"/\"kube-root-ca.crt\"" Nov 03 22:41:47.622525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.622369 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0a7065\"/\"default-dockercfg-z2qtc\"" Nov 03 22:41:47.622525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.622480 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0a7065\"/\"test-model-0a7065-kserve-self-signed-certs\"" Nov 03 22:41:47.629473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.629446 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4"] Nov 03 22:41:47.698240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.698208 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bx8d\" (UniqueName: \"kubernetes.io/projected/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-kube-api-access-9bx8d\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.698448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.698252 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tls-certs\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.698523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.698487 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-model-cache\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.698612 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.698595 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-dshm\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.698675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.698660 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-home\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.698722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.698687 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tmp-dir\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.800686 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.800645 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bx8d\" (UniqueName: \"kubernetes.io/projected/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-kube-api-access-9bx8d\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.800921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.800698 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tls-certs\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.800921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.800864 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-model-cache\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.801065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.800925 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-dshm\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.801065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.801003 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-home\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.801065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.801028 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tmp-dir\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.801348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.801321 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-model-cache\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.801457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.801434 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-home\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.801457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.801448 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tmp-dir\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.803189 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.803159 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-dshm\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.803424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.803403 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tls-certs\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.837217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.837180 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bx8d\" (UniqueName: \"kubernetes.io/projected/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-kube-api-access-9bx8d\") pod \"test-model-0a7065-kserve-5964946f49-k67f4\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:47.915873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:47.915839 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:48.234051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:48.234026 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4"] Nov 03 22:41:48.234342 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:41:48.234281 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ea7032d_33e2_4ea1_ae81_edf2f54d6c0e.slice/crio-2eb322032cb61af36a7bfed7c31c950ab235735216e8b1d4ea88b2cd73e58fad WatchSource:0}: Error finding container 2eb322032cb61af36a7bfed7c31c950ab235735216e8b1d4ea88b2cd73e58fad: Status 404 returned error can't find the container with id 2eb322032cb61af36a7bfed7c31c950ab235735216e8b1d4ea88b2cd73e58fad Nov 03 22:41:48.386977 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:41:48.386943 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:41:48.962185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:48.962138 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" event={"ID":"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e","Type":"ContainerStarted","Data":"8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4"} Nov 03 22:41:48.962185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:48.962183 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" event={"ID":"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e","Type":"ContainerStarted","Data":"2eb322032cb61af36a7bfed7c31c950ab235735216e8b1d4ea88b2cd73e58fad"} Nov 03 22:41:49.001658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:49.001609 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" podStartSLOduration=2.00159594 podStartE2EDuration="2.00159594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:41:48.999199124 +0000 UTC m=+1735.403140102" watchObservedRunningTime="2026-11-03 22:41:49.00159594 +0000 UTC m=+1735.405536915" Nov 03 22:41:49.916890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:49.916856 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:55.119199 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:55.119079 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.188155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:55.186033 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.461319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:55.460369 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.533260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:55.532457 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h" podUID="c282635b-b4c8-40f8-880f-a773dc994c41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.533260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:55.533019 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" podUID="10347389-d227-4891-9a5c-401dec199b97" containerName="main" probeResult="failure" output="Get \"https://10.132.0.207:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.547400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:55.547360 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.823347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:55.823231 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:55.949317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:55.933763 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:56.196580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:56.196524 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" podUID="40bf741a-dbda-425a-94b3-27e46a7d1a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:56.394003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:56.393744 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 22:41:56.400940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:56.400901 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 22:41:57.916506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:57.916467 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:57.929033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:57.929007 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:41:57.969773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:57.969739 4903 scope.go:122] "RemoveContainer" containerID="42ebb07e50eb0b9b65901380f8506a3c13bfb8bc172df9d2863a97cd27ecdd15" Nov 03 22:41:57.981153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:57.981129 4903 scope.go:122] "RemoveContainer" containerID="6524ea3bb37447aed04b9dd1bcde9406a583327ceda8f0a7abead7f0b310c811" Nov 03 22:41:58.150195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:41:58.150168 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:42:04.641317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.641271 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs"] Nov 03 22:42:04.671795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.671755 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs"] Nov 03 22:42:04.673779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.673690 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:04.703269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.701781 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ccdb60-dockercfg-64t45\"" Nov 03 22:42:04.703993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.703969 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ccdb60\"" Nov 03 22:42:04.803366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.803327 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs\" (UID: \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:04.803704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.803652 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhw5z\" (UniqueName: \"kubernetes.io/projected/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-kube-api-access-qhw5z\") pod \"payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs\" (UID: \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:04.905116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.905084 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs\" (UID: \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:04.905284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.905242 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhw5z\" (UniqueName: \"kubernetes.io/projected/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-kube-api-access-qhw5z\") pod \"payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs\" (UID: \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:04.905919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.905901 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs\" (UID: \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:04.926655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:04.926624 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhw5z\" (UniqueName: \"kubernetes.io/projected/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-kube-api-access-qhw5z\") pod \"payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs\" (UID: \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:05.028758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:05.028725 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:05.387741 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:42:05.387698 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6fcce1_a565_4bd4_a769_acbeb168d0d3.slice/crio-f2fecb2f9fc9cc503fd4b1c2e47e86d0bb5150cff09e7c517b939d2d3816bca1 WatchSource:0}: Error finding container f2fecb2f9fc9cc503fd4b1c2e47e86d0bb5150cff09e7c517b939d2d3816bca1: Status 404 returned error can't find the container with id f2fecb2f9fc9cc503fd4b1c2e47e86d0bb5150cff09e7c517b939d2d3816bca1 Nov 03 22:42:05.388072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:05.388049 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs"] Nov 03 22:42:06.224751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:06.224709 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" event={"ID":"ea6fcce1-a565-4bd4-a769-acbeb168d0d3","Type":"ContainerStarted","Data":"0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5"} Nov 03 22:42:06.226263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:06.224781 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" event={"ID":"ea6fcce1-a565-4bd4-a769-acbeb168d0d3","Type":"ContainerStarted","Data":"f2fecb2f9fc9cc503fd4b1c2e47e86d0bb5150cff09e7c517b939d2d3816bca1"} Nov 03 22:42:06.226263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:06.225377 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:06.264122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:06.264069 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" podStartSLOduration=2.264055775 podStartE2EDuration="2.264055775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:06.262527582 +0000 UTC m=+1752.666468559" watchObservedRunningTime="2026-11-03 22:42:06.264055775 +0000 UTC m=+1752.667996750" Nov 03 22:42:08.995548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:08.995503 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778"] Nov 03 22:42:09.000051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.000024 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.021165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.021127 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe39d68-ed57-4205-adea-4098787dae30-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.021426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.021260 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.021426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.021344 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.021426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.021381 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe39d68-ed57-4205-adea-4098787dae30-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.021426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.021410 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.021688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.021442 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.021688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.021628 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.021770 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.021690 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.021770 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.021736 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmjvm\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-kube-api-access-bmjvm\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.024657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.024628 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778"] Nov 03 22:42:09.103965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.103933 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-71a7df-openshift-default-dockercfg-q4mt7\"" Nov 03 22:42:09.122947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.122919 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.122992 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123043 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmjvm\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-kube-api-access-bmjvm\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123134 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe39d68-ed57-4205-adea-4098787dae30-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123239 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123316 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123350 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe39d68-ed57-4205-adea-4098787dae30-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123378 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-certs\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123383 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123461 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123511 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-data\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123778 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.123938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123819 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-credential-socket\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.124019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.123999 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe39d68-ed57-4205-adea-4098787dae30-istiod-ca-cert\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.125584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.125565 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-envoy\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.125931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.125909 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe39d68-ed57-4205-adea-4098787dae30-istio-podinfo\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.145619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.145593 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmjvm\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-kube-api-access-bmjvm\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.145619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.145611 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-istio-token\") pod \"e2e-inf-b-71a7df-openshift-default-74d8445657-fv778\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.319215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.319131 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:09.690219 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:42:09.690186 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe39d68_ed57_4205_adea_4098787dae30.slice/crio-e871f2abdd133a2946262e5d14fe30ba1a677f8aa0202e9ed9ef08d803dd6237 WatchSource:0}: Error finding container e871f2abdd133a2946262e5d14fe30ba1a677f8aa0202e9ed9ef08d803dd6237: Status 404 returned error can't find the container with id e871f2abdd133a2946262e5d14fe30ba1a677f8aa0202e9ed9ef08d803dd6237 Nov 03 22:42:09.690398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.690323 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778"] Nov 03 22:42:09.693375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.693348 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:09.693445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.693411 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:09.693445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:09.693437 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:42:10.261878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:10.261845 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" event={"ID":"afe39d68-ed57-4205-adea-4098787dae30","Type":"ContainerStarted","Data":"db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28"} Nov 03 22:42:10.262278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:10.261929 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" event={"ID":"afe39d68-ed57-4205-adea-4098787dae30","Type":"ContainerStarted","Data":"e871f2abdd133a2946262e5d14fe30ba1a677f8aa0202e9ed9ef08d803dd6237"} Nov 03 22:42:10.318663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:10.318610 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" podStartSLOduration=2.318596306 podStartE2EDuration="2.318596306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:10.314563287 +0000 UTC m=+1756.718504289" watchObservedRunningTime="2026-11-03 22:42:10.318596306 +0000 UTC m=+1756.722537281" Nov 03 22:42:10.319353 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:10.319326 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:10.321225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:10.321201 4903 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 22:42:10.321359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:10.321248 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 22:42:11.320427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:11.320393 4903 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 22:42:11.320828 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:11.320444 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 22:42:12.335907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:12.335426 4903 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 22:42:12.335907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:12.335506 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 22:42:13.319676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:13.319638 4903 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 22:42:13.320026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:13.319983 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 22:42:14.320017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:14.319982 4903 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 22:42:14.320437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:14.320043 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 22:42:15.319477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:15.319441 4903 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 22:42:15.319675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:15.319503 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 22:42:16.324877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:16.324844 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:16.325390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:16.325273 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:16.325788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:16.325768 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:42:17.238787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:17.238754 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:42:18.408572 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:42:18.408527 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:42:29.949815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:29.949777 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs"] Nov 03 22:42:29.954925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:29.954904 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:29.972776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:29.972750 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs"] Nov 03 22:42:29.975496 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:29.975472 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fe7ca0-dockercfg-j4v4m\"" Nov 03 22:42:29.981506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:29.981480 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fe7ca0\"" Nov 03 22:42:30.079633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:30.079577 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb7b18e-b8d1-45dd-99db-999de387124c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs\" (UID: \"bbb7b18e-b8d1-45dd-99db-999de387124c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:30.081048 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:30.081018 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xkz5\" (UniqueName: \"kubernetes.io/projected/bbb7b18e-b8d1-45dd-99db-999de387124c-kube-api-access-5xkz5\") pod \"payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs\" (UID: \"bbb7b18e-b8d1-45dd-99db-999de387124c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:30.222377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:30.218574 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb7b18e-b8d1-45dd-99db-999de387124c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs\" (UID: \"bbb7b18e-b8d1-45dd-99db-999de387124c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:30.222377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:30.218784 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkz5\" (UniqueName: \"kubernetes.io/projected/bbb7b18e-b8d1-45dd-99db-999de387124c-kube-api-access-5xkz5\") pod \"payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs\" (UID: \"bbb7b18e-b8d1-45dd-99db-999de387124c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:30.222377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:30.220025 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb7b18e-b8d1-45dd-99db-999de387124c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs\" (UID: \"bbb7b18e-b8d1-45dd-99db-999de387124c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:30.290570 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:30.290495 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkz5\" (UniqueName: \"kubernetes.io/projected/bbb7b18e-b8d1-45dd-99db-999de387124c-kube-api-access-5xkz5\") pod \"payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs\" (UID: \"bbb7b18e-b8d1-45dd-99db-999de387124c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:30.571056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:30.570969 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:30.919167 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:42:30.919136 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb7b18e_b8d1_45dd_99db_999de387124c.slice/crio-bc88fda8add3534eb4b197eec8c684f4f0f9feb47ecf4d645a2beae27d0ca46f WatchSource:0}: Error finding container bc88fda8add3534eb4b197eec8c684f4f0f9feb47ecf4d645a2beae27d0ca46f: Status 404 returned error can't find the container with id bc88fda8add3534eb4b197eec8c684f4f0f9feb47ecf4d645a2beae27d0ca46f Nov 03 22:42:30.919167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:30.919150 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs"] Nov 03 22:42:31.629683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:31.629647 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" event={"ID":"bbb7b18e-b8d1-45dd-99db-999de387124c","Type":"ContainerStarted","Data":"2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2"} Nov 03 22:42:31.630091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:31.629714 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:31.630091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:31.629737 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" event={"ID":"bbb7b18e-b8d1-45dd-99db-999de387124c","Type":"ContainerStarted","Data":"bc88fda8add3534eb4b197eec8c684f4f0f9feb47ecf4d645a2beae27d0ca46f"} Nov 03 22:42:31.670565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:31.670517 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" podStartSLOduration=2.670502574 podStartE2EDuration="2.670502574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:42:31.666475889 +0000 UTC m=+1778.070416865" watchObservedRunningTime="2026-11-03 22:42:31.670502574 +0000 UTC m=+1778.074443570" Nov 03 22:42:42.641410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:42.641382 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:42:48.246897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:48.246862 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:42:48.524014 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:42:48.523923 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:42:54.136002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:54.135969 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:42:54.136512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:54.136192 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:42:54.145519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:54.145492 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:42:54.146047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:54.146024 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:42:58.016833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:58.016800 4903 scope.go:122] "RemoveContainer" containerID="ff18524c5ee666c3f39309daf110a45401d2a81e2da5ac88b857602c84f1122e" Nov 03 22:42:58.027831 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:42:58.027810 4903 scope.go:122] "RemoveContainer" containerID="df38ef3bf3292b61be3d59590aa04108d2801ff674c500f40f63a271581f011b" Nov 03 22:43:02.899850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:02.899815 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9"] Nov 03 22:43:02.906240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:02.906210 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:02.921743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:02.921695 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9"] Nov 03 22:43:02.926185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:02.926164 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-963936\"" Nov 03 22:43:02.926393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:02.926207 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-963936-dockercfg-xsxzw\"" Nov 03 22:43:03.061517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.061482 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfa9165-ca04-4b2a-a921-5422d1d8428f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9\" (UID: \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:03.061799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.061779 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g58rg\" (UniqueName: \"kubernetes.io/projected/4dfa9165-ca04-4b2a-a921-5422d1d8428f-kube-api-access-g58rg\") pod \"payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9\" (UID: \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:03.090331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.090278 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v"] Nov 03 22:43:03.101561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.101458 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:03.126868 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.126743 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v"] Nov 03 22:43:03.164111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.164073 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfa9165-ca04-4b2a-a921-5422d1d8428f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9\" (UID: \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:03.164486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.164455 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g58rg\" (UniqueName: \"kubernetes.io/projected/4dfa9165-ca04-4b2a-a921-5422d1d8428f-kube-api-access-g58rg\") pod \"payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9\" (UID: \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:03.165184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.165129 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfa9165-ca04-4b2a-a921-5422d1d8428f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9\" (UID: \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:03.202501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.202466 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g58rg\" (UniqueName: \"kubernetes.io/projected/4dfa9165-ca04-4b2a-a921-5422d1d8428f-kube-api-access-g58rg\") pod \"payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9\" (UID: \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:03.232629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.232603 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:03.267739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.267319 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkqk\" (UniqueName: \"kubernetes.io/projected/4b465382-7b48-44d4-9d7d-206d8aba6028-kube-api-access-kfkqk\") pod \"payload-processing-e2e-inf-b-963936-589db57f54-5m29v\" (UID: \"4b465382-7b48-44d4-9d7d-206d8aba6028\") " pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:03.267739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.267530 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b465382-7b48-44d4-9d7d-206d8aba6028-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-963936-589db57f54-5m29v\" (UID: \"4b465382-7b48-44d4-9d7d-206d8aba6028\") " pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:03.369779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.369741 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkqk\" (UniqueName: \"kubernetes.io/projected/4b465382-7b48-44d4-9d7d-206d8aba6028-kube-api-access-kfkqk\") pod \"payload-processing-e2e-inf-b-963936-589db57f54-5m29v\" (UID: \"4b465382-7b48-44d4-9d7d-206d8aba6028\") " pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:03.370030 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.369950 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b465382-7b48-44d4-9d7d-206d8aba6028-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-963936-589db57f54-5m29v\" (UID: \"4b465382-7b48-44d4-9d7d-206d8aba6028\") " pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:03.370931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.370907 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b465382-7b48-44d4-9d7d-206d8aba6028-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-963936-589db57f54-5m29v\" (UID: \"4b465382-7b48-44d4-9d7d-206d8aba6028\") " pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:03.391871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.391842 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkqk\" (UniqueName: \"kubernetes.io/projected/4b465382-7b48-44d4-9d7d-206d8aba6028-kube-api-access-kfkqk\") pod \"payload-processing-e2e-inf-b-963936-589db57f54-5m29v\" (UID: \"4b465382-7b48-44d4-9d7d-206d8aba6028\") " pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:03.457155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.457066 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:03.595425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.591445 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9"] Nov 03 22:43:03.768022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:03.767993 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v"] Nov 03 22:43:03.769493 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:43:03.769465 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b465382_7b48_44d4_9d7d_206d8aba6028.slice/crio-1ad367fa8cd702f0f4e3161f075769d51d35170ed33e3549892203979ea595bb WatchSource:0}: Error finding container 1ad367fa8cd702f0f4e3161f075769d51d35170ed33e3549892203979ea595bb: Status 404 returned error can't find the container with id 1ad367fa8cd702f0f4e3161f075769d51d35170ed33e3549892203979ea595bb Nov 03 22:43:04.093240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:04.093146 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" event={"ID":"4b465382-7b48-44d4-9d7d-206d8aba6028","Type":"ContainerStarted","Data":"5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06"} Nov 03 22:43:04.093240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:04.093208 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" event={"ID":"4b465382-7b48-44d4-9d7d-206d8aba6028","Type":"ContainerStarted","Data":"1ad367fa8cd702f0f4e3161f075769d51d35170ed33e3549892203979ea595bb"} Nov 03 22:43:04.093240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:04.093238 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:04.095040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:04.095015 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" event={"ID":"4dfa9165-ca04-4b2a-a921-5422d1d8428f","Type":"ContainerStarted","Data":"df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7"} Nov 03 22:43:04.095166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:04.095062 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" event={"ID":"4dfa9165-ca04-4b2a-a921-5422d1d8428f","Type":"ContainerStarted","Data":"a99608722488be4a875ebefbe600bd391c87260ad10f0f102e28204b8bb17443"} Nov 03 22:43:04.095166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:04.095092 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:04.131320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:04.131195 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" podStartSLOduration=2.131177366 podStartE2EDuration="2.131177366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:04.131059315 +0000 UTC m=+1810.535000293" watchObservedRunningTime="2026-11-03 22:43:04.131177366 +0000 UTC m=+1810.535118333" Nov 03 22:43:04.170330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:04.170238 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" podStartSLOduration=2.170221862 podStartE2EDuration="2.170221862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:04.168135091 +0000 UTC m=+1810.572076069" watchObservedRunningTime="2026-11-03 22:43:04.170221862 +0000 UTC m=+1810.574162836" Nov 03 22:43:05.602713 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.602675 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8"] Nov 03 22:43:05.609662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.609637 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.626182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.626144 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8"] Nov 03 22:43:05.658944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.658913 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-124490-openshift-default-dockercfg-p9v8q\"" Nov 03 22:43:05.782455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.778074 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-credential-socket\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.782455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.778138 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb281e27-8506-4de9-91d3-4800a8f40dc3-istiod-ca-cert\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.782455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.778223 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-envoy\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.782455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.778392 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-podinfo\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.782455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.778540 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-token\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.782455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.778601 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhwb\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-kube-api-access-jzhwb\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.782455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.778641 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-certs\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.782455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.778671 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-data\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.782455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.778712 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-socket\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.886520 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-credential-socket\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.886569 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb281e27-8506-4de9-91d3-4800a8f40dc3-istiod-ca-cert\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.886629 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-envoy\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.886795 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-podinfo\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.886903 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-token\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.886943 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhwb\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-kube-api-access-jzhwb\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.886987 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-certs\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.887011 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-data\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.887042 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-socket\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.890382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.890406 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-credential-socket\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.891259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.891002 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-certs\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.891567 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.891537 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-socket\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.893759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.893729 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb281e27-8506-4de9-91d3-4800a8f40dc3-istiod-ca-cert\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.893992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.893973 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-data\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.894631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.894600 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-podinfo\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.899593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.899545 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-envoy\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.917312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.916288 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-token\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.920834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.919288 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhwb\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-kube-api-access-jzhwb\") pod \"e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:05.935773 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:05.934589 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:06.295657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:06.295627 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8"] Nov 03 22:43:06.297121 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:43:06.297099 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb281e27_8506_4de9_91d3_4800a8f40dc3.slice/crio-56a23ee9253a861968e08e675d6decb6cc4ffde1b25a92fdfd6d134ed05896d4 WatchSource:0}: Error finding container 56a23ee9253a861968e08e675d6decb6cc4ffde1b25a92fdfd6d134ed05896d4: Status 404 returned error can't find the container with id 56a23ee9253a861968e08e675d6decb6cc4ffde1b25a92fdfd6d134ed05896d4 Nov 03 22:43:06.301174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:06.301131 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:43:06.301284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:06.301220 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:43:06.301284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:06.301260 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:43:07.131684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:07.131649 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" event={"ID":"eb281e27-8506-4de9-91d3-4800a8f40dc3","Type":"ContainerStarted","Data":"afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b"} Nov 03 22:43:07.132145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:07.131705 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" event={"ID":"eb281e27-8506-4de9-91d3-4800a8f40dc3","Type":"ContainerStarted","Data":"56a23ee9253a861968e08e675d6decb6cc4ffde1b25a92fdfd6d134ed05896d4"} Nov 03 22:43:07.188201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:07.188138 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" podStartSLOduration=2.188120554 podStartE2EDuration="2.188120554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:07.183525577 +0000 UTC m=+1813.587466553" watchObservedRunningTime="2026-11-03 22:43:07.188120554 +0000 UTC m=+1813.592061530" Nov 03 22:43:07.935363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:07.935322 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:07.942019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:07.941989 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:08.147365 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:08.147328 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:08.148415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:08.148392 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:43:11.252014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.251980 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs"] Nov 03 22:43:11.257348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.257323 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:11.274439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.274411 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs"] Nov 03 22:43:11.294860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.294829 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-124490\"" Nov 03 22:43:11.473320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.467173 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgbwj\" (UniqueName: \"kubernetes.io/projected/4644ccf9-0c96-45ef-8892-07da96b7c784-kube-api-access-qgbwj\") pod \"payload-processing-e2e-inf-b-124490-77db798586-7ghqs\" (UID: \"4644ccf9-0c96-45ef-8892-07da96b7c784\") " pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:11.473320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.467646 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644ccf9-0c96-45ef-8892-07da96b7c784-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-124490-77db798586-7ghqs\" (UID: \"4644ccf9-0c96-45ef-8892-07da96b7c784\") " pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:11.575369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.574522 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgbwj\" (UniqueName: \"kubernetes.io/projected/4644ccf9-0c96-45ef-8892-07da96b7c784-kube-api-access-qgbwj\") pod \"payload-processing-e2e-inf-b-124490-77db798586-7ghqs\" (UID: \"4644ccf9-0c96-45ef-8892-07da96b7c784\") " pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:11.575369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.574873 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644ccf9-0c96-45ef-8892-07da96b7c784-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-124490-77db798586-7ghqs\" (UID: \"4644ccf9-0c96-45ef-8892-07da96b7c784\") " pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:11.579313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.576014 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644ccf9-0c96-45ef-8892-07da96b7c784-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-124490-77db798586-7ghqs\" (UID: \"4644ccf9-0c96-45ef-8892-07da96b7c784\") " pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:11.627651 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.625880 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgbwj\" (UniqueName: \"kubernetes.io/projected/4644ccf9-0c96-45ef-8892-07da96b7c784-kube-api-access-qgbwj\") pod \"payload-processing-e2e-inf-b-124490-77db798586-7ghqs\" (UID: \"4644ccf9-0c96-45ef-8892-07da96b7c784\") " pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:11.877050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:11.876960 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:12.190012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:12.189976 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs"] Nov 03 22:43:12.195403 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:43:12.195373 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4644ccf9_0c96_45ef_8892_07da96b7c784.slice/crio-3751261e184b0874d7a1800fffb38714dc71cdac5b8538d27c778bdca244bb4f WatchSource:0}: Error finding container 3751261e184b0874d7a1800fffb38714dc71cdac5b8538d27c778bdca244bb4f: Status 404 returned error can't find the container with id 3751261e184b0874d7a1800fffb38714dc71cdac5b8538d27c778bdca244bb4f Nov 03 22:43:12.806433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:12.806395 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff"] Nov 03 22:43:12.806925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:12.806640 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerName="istio-proxy" containerID="cri-o://32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561" gracePeriod=30 Nov 03 22:43:13.208218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:13.208179 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" event={"ID":"4644ccf9-0c96-45ef-8892-07da96b7c784","Type":"ContainerStarted","Data":"ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5"} Nov 03 22:43:13.208458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:13.208240 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" event={"ID":"4644ccf9-0c96-45ef-8892-07da96b7c784","Type":"ContainerStarted","Data":"3751261e184b0874d7a1800fffb38714dc71cdac5b8538d27c778bdca244bb4f"} Nov 03 22:43:13.208458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:13.208270 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:13.246686 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:13.246621 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" podStartSLOduration=2.246604228 podStartE2EDuration="2.246604228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:13.245520875 +0000 UTC m=+1819.649461851" watchObservedRunningTime="2026-11-03 22:43:13.246604228 +0000 UTC m=+1819.650545208" Nov 03 22:43:15.118318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:15.118263 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:43:15.122757 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:15.122731 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:43:17.448716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:17.448683 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj"] Nov 03 22:43:17.449146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:17.448875 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" containerID="cri-o://be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b" gracePeriod=30 Nov 03 22:43:18.106796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.106767 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:43:18.208612 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.208577 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-envoy\") pod \"1395a56f-875d-4faa-a8a6-c1caf5438efe\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " Nov 03 22:43:18.208805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.208724 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-credential-socket\") pod \"1395a56f-875d-4faa-a8a6-c1caf5438efe\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " Nov 03 22:43:18.208805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.208780 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-data\") pod \"1395a56f-875d-4faa-a8a6-c1caf5438efe\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " Nov 03 22:43:18.208918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.208825 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-kube-api-access-qxjj7\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-kube-api-access-qxjj7\") pod \"1395a56f-875d-4faa-a8a6-c1caf5438efe\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " Nov 03 22:43:18.208980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.208967 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-token\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-token\") pod \"1395a56f-875d-4faa-a8a6-c1caf5438efe\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " Nov 03 22:43:18.209034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.208998 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1395a56f-875d-4faa-a8a6-c1caf5438efe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1395a56f-875d-4faa-a8a6-c1caf5438efe-istiod-ca-cert\") pod \"1395a56f-875d-4faa-a8a6-c1caf5438efe\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " Nov 03 22:43:18.209034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.209027 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-certs\") pod \"1395a56f-875d-4faa-a8a6-c1caf5438efe\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " Nov 03 22:43:18.209191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.209173 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-socket\") pod \"1395a56f-875d-4faa-a8a6-c1caf5438efe\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " Nov 03 22:43:18.209252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.209192 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-credential-socket" pod "1395a56f-875d-4faa-a8a6-c1caf5438efe" (UID: "1395a56f-875d-4faa-a8a6-c1caf5438efe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:18.209593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.209375 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-podinfo\") pod \"1395a56f-875d-4faa-a8a6-c1caf5438efe\" (UID: \"1395a56f-875d-4faa-a8a6-c1caf5438efe\") " Nov 03 22:43:18.209593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.209417 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-data" pod "1395a56f-875d-4faa-a8a6-c1caf5438efe" (UID: "1395a56f-875d-4faa-a8a6-c1caf5438efe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:18.209767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.209630 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-certs" pod "1395a56f-875d-4faa-a8a6-c1caf5438efe" (UID: "1395a56f-875d-4faa-a8a6-c1caf5438efe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:18.209947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.209922 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1395a56f-875d-4faa-a8a6-c1caf5438efe-istiod-ca-cert" pod "1395a56f-875d-4faa-a8a6-c1caf5438efe" (UID: "1395a56f-875d-4faa-a8a6-c1caf5438efe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:18.210306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.210264 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:18.210408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.210309 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:18.210408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.210325 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1395a56f-875d-4faa-a8a6-c1caf5438efe-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:18.210408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.210338 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:18.210621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.210585 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-socket" pod "1395a56f-875d-4faa-a8a6-c1caf5438efe" (UID: "1395a56f-875d-4faa-a8a6-c1caf5438efe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:18.212805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.212777 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-envoy" pod "1395a56f-875d-4faa-a8a6-c1caf5438efe" (UID: "1395a56f-875d-4faa-a8a6-c1caf5438efe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:18.212910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.212878 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-podinfo" pod "1395a56f-875d-4faa-a8a6-c1caf5438efe" (UID: "1395a56f-875d-4faa-a8a6-c1caf5438efe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:18.215591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.215559 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-kube-api-access-qxjj7" pod "1395a56f-875d-4faa-a8a6-c1caf5438efe" (UID: "1395a56f-875d-4faa-a8a6-c1caf5438efe"). InnerVolumeSpecName "kube-api-access-qxjj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:18.215826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.215799 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-token" pod "1395a56f-875d-4faa-a8a6-c1caf5438efe" (UID: "1395a56f-875d-4faa-a8a6-c1caf5438efe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:18.292306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.292024 4903 generic.go:356] "Generic (PLEG): container finished" podID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerID="32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561" exitCode=0 Nov 03 22:43:18.292306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.292065 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" event={"ID":"1395a56f-875d-4faa-a8a6-c1caf5438efe","Type":"ContainerDied","Data":"32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561"} Nov 03 22:43:18.292306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.292125 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" event={"ID":"1395a56f-875d-4faa-a8a6-c1caf5438efe","Type":"ContainerDied","Data":"8f02afd199c6e329b238d89866454c6ca78e49dda9d527034f1e9022487f4a39"} Nov 03 22:43:18.292306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.292147 4903 scope.go:122] "RemoveContainer" containerID="32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561" Nov 03 22:43:18.292306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.292080 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff" Nov 03 22:43:18.307121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.307103 4903 scope.go:122] "RemoveContainer" containerID="32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561" Nov 03 22:43:18.307483 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:43:18.307453 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561\": container with ID starting with 32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561 not found: ID does not exist" containerID="32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561" Nov 03 22:43:18.307597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.307487 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561"} err="failed to get container status \"32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561\": rpc error: code = NotFound desc = could not find container \"32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561\": container with ID starting with 32f50d608e5391b6dc0ffb120f470b8a75342f6b2916dc623268e5e805886561 not found: ID does not exist" Nov 03 22:43:18.311827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.311804 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:18.311827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.311828 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:18.311827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.311837 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:18.312026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.311846 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxjj7\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-kube-api-access-qxjj7\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:18.312026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.311856 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1395a56f-875d-4faa-a8a6-c1caf5438efe-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:18.344676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.344613 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff"] Nov 03 22:43:18.349128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:18.349108 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-jpkff"] Nov 03 22:43:18.385710 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:43:18.385680 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:43:19.738827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:19.738788 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" path="/var/lib/kubelet/pods/1395a56f-875d-4faa-a8a6-c1caf5438efe/volumes" Nov 03 22:43:22.747901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.747876 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:43:22.793025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.792993 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-socket\") pod \"7a858a6a-0039-46e3-b77f-531dcaefb894\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " Nov 03 22:43:22.793202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793036 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-token\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-token\") pod \"7a858a6a-0039-46e3-b77f-531dcaefb894\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " Nov 03 22:43:22.793202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793064 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a858a6a-0039-46e3-b77f-531dcaefb894-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a858a6a-0039-46e3-b77f-531dcaefb894-istiod-ca-cert\") pod \"7a858a6a-0039-46e3-b77f-531dcaefb894\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " Nov 03 22:43:22.793202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793086 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-kube-api-access-6gghp\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-kube-api-access-6gghp\") pod \"7a858a6a-0039-46e3-b77f-531dcaefb894\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " Nov 03 22:43:22.793379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793212 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-podinfo\") pod \"7a858a6a-0039-46e3-b77f-531dcaefb894\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " Nov 03 22:43:22.793379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793271 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-socket" pod "7a858a6a-0039-46e3-b77f-531dcaefb894" (UID: "7a858a6a-0039-46e3-b77f-531dcaefb894"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.793471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793380 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-data\") pod \"7a858a6a-0039-46e3-b77f-531dcaefb894\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " Nov 03 22:43:22.793535 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793487 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-envoy\") pod \"7a858a6a-0039-46e3-b77f-531dcaefb894\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " Nov 03 22:43:22.793614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793594 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a858a6a-0039-46e3-b77f-531dcaefb894-istiod-ca-cert" pod "7a858a6a-0039-46e3-b77f-531dcaefb894" (UID: "7a858a6a-0039-46e3-b77f-531dcaefb894"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:43:22.793697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793596 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-certs\") pod \"7a858a6a-0039-46e3-b77f-531dcaefb894\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " Nov 03 22:43:22.793755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793699 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-data" pod "7a858a6a-0039-46e3-b77f-531dcaefb894" (UID: "7a858a6a-0039-46e3-b77f-531dcaefb894"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.793833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793813 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-certs" pod "7a858a6a-0039-46e3-b77f-531dcaefb894" (UID: "7a858a6a-0039-46e3-b77f-531dcaefb894"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.793911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.793891 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-credential-socket\") pod \"7a858a6a-0039-46e3-b77f-531dcaefb894\" (UID: \"7a858a6a-0039-46e3-b77f-531dcaefb894\") " Nov 03 22:43:22.794153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.794129 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-credential-socket" pod "7a858a6a-0039-46e3-b77f-531dcaefb894" (UID: "7a858a6a-0039-46e3-b77f-531dcaefb894"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.794884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.794862 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.794975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.794889 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.794975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.794902 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.794975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.794918 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.794975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.794932 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a858a6a-0039-46e3-b77f-531dcaefb894-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.795650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.795627 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-podinfo" pod "7a858a6a-0039-46e3-b77f-531dcaefb894" (UID: "7a858a6a-0039-46e3-b77f-531dcaefb894"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:43:22.795748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.795676 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-envoy" pod "7a858a6a-0039-46e3-b77f-531dcaefb894" (UID: "7a858a6a-0039-46e3-b77f-531dcaefb894"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:43:22.795866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.795849 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-token" pod "7a858a6a-0039-46e3-b77f-531dcaefb894" (UID: "7a858a6a-0039-46e3-b77f-531dcaefb894"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:22.796650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.796622 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-kube-api-access-6gghp" pod "7a858a6a-0039-46e3-b77f-531dcaefb894" (UID: "7a858a6a-0039-46e3-b77f-531dcaefb894"). InnerVolumeSpecName "kube-api-access-6gghp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:43:22.896841 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.896811 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.896841 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.896839 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.897056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.896858 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gghp\" (UniqueName: \"kubernetes.io/projected/7a858a6a-0039-46e3-b77f-531dcaefb894-kube-api-access-6gghp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:22.897056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:22.896887 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a858a6a-0039-46e3-b77f-531dcaefb894-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:43:23.348185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.348149 4903 generic.go:356] "Generic (PLEG): container finished" podID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerID="be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b" exitCode=0 Nov 03 22:43:23.348363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.348221 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" Nov 03 22:43:23.348363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.348240 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" event={"ID":"7a858a6a-0039-46e3-b77f-531dcaefb894","Type":"ContainerDied","Data":"be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b"} Nov 03 22:43:23.348363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.348324 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj" event={"ID":"7a858a6a-0039-46e3-b77f-531dcaefb894","Type":"ContainerDied","Data":"3e5221161ac929d7e1cf9aeec6c5bcd72bbfec9db162de0b3a3c7813b25513e2"} Nov 03 22:43:23.348363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.348348 4903 scope.go:122] "RemoveContainer" containerID="be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b" Nov 03 22:43:23.361398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.361364 4903 scope.go:122] "RemoveContainer" containerID="be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b" Nov 03 22:43:23.361665 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:43:23.361646 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b\": container with ID starting with be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b not found: ID does not exist" containerID="be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b" Nov 03 22:43:23.361721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.361673 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b"} err="failed to get container status \"be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b\": rpc error: code = NotFound desc = could not find container \"be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b\": container with ID starting with be26a368274519744622a3c3cf98be08ee2f27bb9c425521bdbcc6be8246787b not found: ID does not exist" Nov 03 22:43:23.417144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.417110 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj"] Nov 03 22:43:23.425654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.425625 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-bsswj"] Nov 03 22:43:23.738222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:23.738191 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" path="/var/lib/kubelet/pods/7a858a6a-0039-46e3-b77f-531dcaefb894/volumes" Nov 03 22:43:24.220245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:24.220219 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:43:39.189077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.188986 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2"] Nov 03 22:43:39.190539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.190512 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" Nov 03 22:43:39.190539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.190539 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" Nov 03 22:43:39.190759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.190597 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerName="istio-proxy" Nov 03 22:43:39.190759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.190606 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerName="istio-proxy" Nov 03 22:43:39.190864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.190816 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a858a6a-0039-46e3-b77f-531dcaefb894" containerName="istio-proxy" Nov 03 22:43:39.190864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.190836 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1395a56f-875d-4faa-a8a6-c1caf5438efe" containerName="istio-proxy" Nov 03 22:43:39.197486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.197461 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.222523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.222348 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7940ea\"/\"openshift-service-ca.crt\"" Nov 03 22:43:39.222523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.222378 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7940ea\"/\"kube-root-ca.crt\"" Nov 03 22:43:39.222523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.222479 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7940ea\"/\"default-dockercfg-hsgvr\"" Nov 03 22:43:39.223626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.223605 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7940ea\"/\"test-model-7940ea-kserve-self-signed-certs\"" Nov 03 22:43:39.223998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.223916 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2"] Nov 03 22:43:39.340380 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.340350 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-dshm\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.340557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.340407 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-kube-api-access-hkncd\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.340692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.340668 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-home\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.340736 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.340708 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tmp-dir\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.340815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.340799 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tls-certs\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.340880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.340863 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-model-cache\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.442207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.442124 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-dshm\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.442207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.442174 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-kube-api-access-hkncd\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.442501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.442355 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-home\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.442501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.442376 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tmp-dir\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.442501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.442413 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tls-certs\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.442501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.442468 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-model-cache\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.443260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.442685 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-home\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.443260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.442758 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tmp-dir\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.443260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.442860 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-model-cache\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.444552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.444529 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-dshm\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.444869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.444850 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tls-certs\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.464213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.464184 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-kube-api-access-hkncd\") pod \"test-model-7940ea-kserve-7765dc7479-6r5p2\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.527323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.526966 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:39.877365 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:43:39.877333 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea71a7f5_2351_4704_80b7_8cb2b943a3ca.slice/crio-240d102bc20494a2f077f3aadfcfd95d208adbefd462873e0b81ca8e4ba5ae5d WatchSource:0}: Error finding container 240d102bc20494a2f077f3aadfcfd95d208adbefd462873e0b81ca8e4ba5ae5d: Status 404 returned error can't find the container with id 240d102bc20494a2f077f3aadfcfd95d208adbefd462873e0b81ca8e4ba5ae5d Nov 03 22:43:39.878281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:39.878255 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2"] Nov 03 22:43:40.536164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:40.536127 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" event={"ID":"ea71a7f5-2351-4704-80b7-8cb2b943a3ca","Type":"ContainerStarted","Data":"5f3e5e1f8c801a493f14785ce06fc94e8be505ef5c06d4fda58dc11b9fc01f51"} Nov 03 22:43:40.536602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:40.536178 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" event={"ID":"ea71a7f5-2351-4704-80b7-8cb2b943a3ca","Type":"ContainerStarted","Data":"240d102bc20494a2f077f3aadfcfd95d208adbefd462873e0b81ca8e4ba5ae5d"} Nov 03 22:43:40.577991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:40.577940 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" podStartSLOduration=1.5779255509999999 podStartE2EDuration="1.577925551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:43:40.574906283 +0000 UTC m=+1846.978847259" watchObservedRunningTime="2026-11-03 22:43:40.577925551 +0000 UTC m=+1846.981866572" Nov 03 22:43:41.535313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:41.531715 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:48.386735 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:43:48.386700 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:43:49.527983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:49.527916 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:49.546037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:49.546010 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:49.629097 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:49.629066 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:43:58.067579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:58.067538 4903 scope.go:122] "RemoveContainer" containerID="fb4b75e7dff2f988c2b296004892297b414e9966eed44ed5273bab05271ad68e" Nov 03 22:43:58.081578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:58.081458 4903 scope.go:122] "RemoveContainer" containerID="f7f9a444966779560bd7e61f9223e1bf4150cfec88bce66e73a12f404cb378fb" Nov 03 22:43:59.081314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:43:59.071498 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:44:12.880663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.880632 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx"] Nov 03 22:44:12.887827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.887802 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:12.911837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.911807 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx"] Nov 03 22:44:12.917981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.917948 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ccdb60\"/\"test-model-ccdb60-kserve-self-signed-certs\"" Nov 03 22:44:12.918140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.918069 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ccdb60\"/\"openshift-service-ca.crt\"" Nov 03 22:44:12.918140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.918130 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ccdb60\"/\"default-dockercfg-rf688\"" Nov 03 22:44:12.918259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.918206 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ccdb60\"/\"kube-root-ca.crt\"" Nov 03 22:44:12.961195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.961155 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-tmp-dir\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:12.961414 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.961262 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-dshm\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:12.961414 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.961335 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-model-cache\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:12.961414 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.961368 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgsk\" (UniqueName: \"kubernetes.io/projected/481dd564-4ea1-48d3-8266-56fa2fb76b75-kube-api-access-fcgsk\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:12.961546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.961529 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481dd564-4ea1-48d3-8266-56fa2fb76b75-tls-certs\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:12.961582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:12.961564 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-home\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.062744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.062711 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481dd564-4ea1-48d3-8266-56fa2fb76b75-tls-certs\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.062931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.062764 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-home\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.062931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.062872 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-tmp-dir\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.062931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.062917 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-dshm\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.063098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.062936 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-model-cache\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.063098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.062966 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgsk\" (UniqueName: \"kubernetes.io/projected/481dd564-4ea1-48d3-8266-56fa2fb76b75-kube-api-access-fcgsk\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.063281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.063230 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-home\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.063422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.063280 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-model-cache\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.063422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.063374 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-tmp-dir\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.065174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.065147 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-dshm\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.065555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.065538 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481dd564-4ea1-48d3-8266-56fa2fb76b75-tls-certs\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.084100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.084079 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgsk\" (UniqueName: \"kubernetes.io/projected/481dd564-4ea1-48d3-8266-56fa2fb76b75-kube-api-access-fcgsk\") pod \"test-model-ccdb60-kserve-655f55459-dvvqx\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.224895 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.224859 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:13.616554 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:44:13.616526 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod481dd564_4ea1_48d3_8266_56fa2fb76b75.slice/crio-b279affbc2a80d0d895d3f0d6318db683579ce22f60c87457be105f5a72c198e WatchSource:0}: Error finding container b279affbc2a80d0d895d3f0d6318db683579ce22f60c87457be105f5a72c198e: Status 404 returned error can't find the container with id b279affbc2a80d0d895d3f0d6318db683579ce22f60c87457be105f5a72c198e Nov 03 22:44:13.618517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.618493 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx"] Nov 03 22:44:13.951632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.951595 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" event={"ID":"481dd564-4ea1-48d3-8266-56fa2fb76b75","Type":"ContainerStarted","Data":"eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9"} Nov 03 22:44:13.951632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.951639 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" event={"ID":"481dd564-4ea1-48d3-8266-56fa2fb76b75","Type":"ContainerStarted","Data":"b279affbc2a80d0d895d3f0d6318db683579ce22f60c87457be105f5a72c198e"} Nov 03 22:44:13.994166 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:13.994112 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" podStartSLOduration=1.99409772 podStartE2EDuration="1.99409772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:13.988857387 +0000 UTC m=+1880.392798363" watchObservedRunningTime="2026-11-03 22:44:13.99409772 +0000 UTC m=+1880.398038695" Nov 03 22:44:14.225708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:14.225630 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:18.390061 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:44:18.390026 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:44:23.226002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:23.225965 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:23.239437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:23.239410 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:24.051800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:24.051768 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:44:26.385201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.385167 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf"] Nov 03 22:44:26.390549 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.390526 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.404653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.404619 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:44:26.406464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.406442 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf"] Nov 03 22:44:26.411422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.411401 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:26.411511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.411430 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:44:26.414948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.414923 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:44:26.526965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.526929 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4512c7f-ctzhf\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.527162 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.527071 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqgnf\" (UniqueName: \"kubernetes.io/projected/08c41be0-54fb-458c-9c04-6096c4e589ba-kube-api-access-tqgnf\") pod \"maas-api-revoke-sub-a4512c7f-ctzhf\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.527162 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.527148 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4512c7f-ctzhf\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.629400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.629363 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqgnf\" (UniqueName: \"kubernetes.io/projected/08c41be0-54fb-458c-9c04-6096c4e589ba-kube-api-access-tqgnf\") pod \"maas-api-revoke-sub-a4512c7f-ctzhf\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.629597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.629409 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4512c7f-ctzhf\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.629734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.629714 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4512c7f-ctzhf\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.630321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.630282 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4512c7f-ctzhf\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.631815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.631795 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4512c7f-ctzhf\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.650967 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.650945 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqgnf\" (UniqueName: \"kubernetes.io/projected/08c41be0-54fb-458c-9c04-6096c4e589ba-kube-api-access-tqgnf\") pod \"maas-api-revoke-sub-a4512c7f-ctzhf\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:26.708963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:26.708935 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:27.024877 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:44:27.024847 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c41be0_54fb_458c_9c04_6096c4e589ba.slice/crio-ef3d18cef917af00298082a236b6c462329ad8a8cedf782cc342089c0d35bd0b WatchSource:0}: Error finding container ef3d18cef917af00298082a236b6c462329ad8a8cedf782cc342089c0d35bd0b: Status 404 returned error can't find the container with id ef3d18cef917af00298082a236b6c462329ad8a8cedf782cc342089c0d35bd0b Nov 03 22:44:27.025119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:27.025097 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf"] Nov 03 22:44:27.071637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:27.071610 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" event={"ID":"08c41be0-54fb-458c-9c04-6096c4e589ba","Type":"ContainerStarted","Data":"ef3d18cef917af00298082a236b6c462329ad8a8cedf782cc342089c0d35bd0b"} Nov 03 22:44:28.081683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:28.081648 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" event={"ID":"08c41be0-54fb-458c-9c04-6096c4e589ba","Type":"ContainerStarted","Data":"eb69a4883811092986f6f82b2a2fa2a7f2c8ba37b87932b25b20ad74cecd8370"} Nov 03 22:44:28.121307 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:28.121250 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" podStartSLOduration=2.12123587 podStartE2EDuration="2.12123587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:28.117725274 +0000 UTC m=+1894.521666249" watchObservedRunningTime="2026-11-03 22:44:28.12123587 +0000 UTC m=+1894.525176845" Nov 03 22:44:29.093028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:29.092993 4903 generic.go:356] "Generic (PLEG): container finished" podID="08c41be0-54fb-458c-9c04-6096c4e589ba" containerID="eb69a4883811092986f6f82b2a2fa2a7f2c8ba37b87932b25b20ad74cecd8370" exitCode=0 Nov 03 22:44:29.093434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:29.093075 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" event={"ID":"08c41be0-54fb-458c-9c04-6096c4e589ba","Type":"ContainerDied","Data":"eb69a4883811092986f6f82b2a2fa2a7f2c8ba37b87932b25b20ad74cecd8370"} Nov 03 22:44:30.324962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.324939 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:30.500733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.500702 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c41be0-54fb-458c-9c04-6096c4e589ba-kube-api-access-tqgnf\" (UniqueName: \"kubernetes.io/projected/08c41be0-54fb-458c-9c04-6096c4e589ba-kube-api-access-tqgnf\") pod \"08c41be0-54fb-458c-9c04-6096c4e589ba\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " Nov 03 22:44:30.500919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.500811 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-openshift-ca\") pod \"08c41be0-54fb-458c-9c04-6096c4e589ba\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " Nov 03 22:44:30.501201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.501176 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-xks-ca\") pod \"08c41be0-54fb-458c-9c04-6096c4e589ba\" (UID: \"08c41be0-54fb-458c-9c04-6096c4e589ba\") " Nov 03 22:44:30.501201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.501194 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-openshift-ca" pod "08c41be0-54fb-458c-9c04-6096c4e589ba" (UID: "08c41be0-54fb-458c-9c04-6096c4e589ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:30.501779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.501759 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:30.502935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.502906 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c41be0-54fb-458c-9c04-6096c4e589ba-kube-api-access-tqgnf" pod "08c41be0-54fb-458c-9c04-6096c4e589ba" (UID: "08c41be0-54fb-458c-9c04-6096c4e589ba"). InnerVolumeSpecName "kube-api-access-tqgnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:30.503104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.503085 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-xks-ca" pod "08c41be0-54fb-458c-9c04-6096c4e589ba" (UID: "08c41be0-54fb-458c-9c04-6096c4e589ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:30.603554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.603520 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c41be0-54fb-458c-9c04-6096c4e589ba-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:30.603554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:30.603549 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqgnf\" (UniqueName: \"kubernetes.io/projected/08c41be0-54fb-458c-9c04-6096c4e589ba-kube-api-access-tqgnf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:31.112457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:31.112426 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" Nov 03 22:44:31.112661 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:31.112458 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf" event={"ID":"08c41be0-54fb-458c-9c04-6096c4e589ba","Type":"ContainerDied","Data":"ef3d18cef917af00298082a236b6c462329ad8a8cedf782cc342089c0d35bd0b"} Nov 03 22:44:31.112661 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:31.112506 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef3d18cef917af00298082a236b6c462329ad8a8cedf782cc342089c0d35bd0b" Nov 03 22:44:47.975100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:47.975060 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq"] Nov 03 22:44:47.975606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:47.975260 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" containerID="cri-o://6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906" gracePeriod=30 Nov 03 22:44:48.174039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.174002 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2"] Nov 03 22:44:48.400403 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:44:48.400227 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:44:48.693826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.693742 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp"] Nov 03 22:44:48.694630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.694614 4903 cpu_manager.go:414] "containerMap: removing container" podUID="08c41be0-54fb-458c-9c04-6096c4e589ba" containerName="revoke-keys" Nov 03 22:44:48.694698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.694632 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c41be0-54fb-458c-9c04-6096c4e589ba" containerName="revoke-keys" Nov 03 22:44:48.694848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.694837 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c41be0-54fb-458c-9c04-6096c4e589ba" containerName="revoke-keys" Nov 03 22:44:48.698940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.698920 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:48.719446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.719415 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp"] Nov 03 22:44:48.721026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.721003 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:44:48.721132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.721099 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:44:48.727308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.721910 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:44:48.727308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.722040 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:44:48.777871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.777841 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/300945fb-de39-48f9-982f-566890adfb48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0e6815-s22hp\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:48.778120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.778100 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m46ww\" (UniqueName: \"kubernetes.io/projected/300945fb-de39-48f9-982f-566890adfb48-kube-api-access-m46ww\") pod \"maas-api-revoke-sub-0a0e6815-s22hp\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:48.778261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.778245 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/300945fb-de39-48f9-982f-566890adfb48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0e6815-s22hp\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:48.879454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.879422 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m46ww\" (UniqueName: \"kubernetes.io/projected/300945fb-de39-48f9-982f-566890adfb48-kube-api-access-m46ww\") pod \"maas-api-revoke-sub-0a0e6815-s22hp\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:48.879653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.879531 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/300945fb-de39-48f9-982f-566890adfb48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0e6815-s22hp\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:48.879653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.879596 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/300945fb-de39-48f9-982f-566890adfb48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0e6815-s22hp\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:48.880349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.880323 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/300945fb-de39-48f9-982f-566890adfb48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a0e6815-s22hp\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:48.882015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.881990 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/300945fb-de39-48f9-982f-566890adfb48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a0e6815-s22hp\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:48.900560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:48.900539 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m46ww\" (UniqueName: \"kubernetes.io/projected/300945fb-de39-48f9-982f-566890adfb48-kube-api-access-m46ww\") pod \"maas-api-revoke-sub-0a0e6815-s22hp\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:49.025370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:49.025284 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:49.337413 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:44:49.337376 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300945fb_de39_48f9_982f_566890adfb48.slice/crio-49a924b58894e3fb457a4ac1d7652126f037dadd2c2eef1894dae9c3093acd0d WatchSource:0}: Error finding container 49a924b58894e3fb457a4ac1d7652126f037dadd2c2eef1894dae9c3093acd0d: Status 404 returned error can't find the container with id 49a924b58894e3fb457a4ac1d7652126f037dadd2c2eef1894dae9c3093acd0d Nov 03 22:44:49.337744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:49.337723 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp"] Nov 03 22:44:49.353557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:49.353527 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" event={"ID":"300945fb-de39-48f9-982f-566890adfb48","Type":"ContainerStarted","Data":"49a924b58894e3fb457a4ac1d7652126f037dadd2c2eef1894dae9c3093acd0d"} Nov 03 22:44:50.365022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:50.364982 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" event={"ID":"300945fb-de39-48f9-982f-566890adfb48","Type":"ContainerStarted","Data":"a41f82e17a7da479a33f754f6251e9a8a63fc3a4bab77acb7e8e39e06126ff9d"} Nov 03 22:44:50.402951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:50.402894 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" podStartSLOduration=2.402878036 podStartE2EDuration="2.402878036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:50.401121549 +0000 UTC m=+1916.805062525" watchObservedRunningTime="2026-11-03 22:44:50.402878036 +0000 UTC m=+1916.806819063" Nov 03 22:44:51.374986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.374948 4903 generic.go:356] "Generic (PLEG): container finished" podID="300945fb-de39-48f9-982f-566890adfb48" containerID="a41f82e17a7da479a33f754f6251e9a8a63fc3a4bab77acb7e8e39e06126ff9d" exitCode=0 Nov 03 22:44:51.375403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.375033 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" event={"ID":"300945fb-de39-48f9-982f-566890adfb48","Type":"ContainerDied","Data":"a41f82e17a7da479a33f754f6251e9a8a63fc3a4bab77acb7e8e39e06126ff9d"} Nov 03 22:44:51.406505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.406472 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw"] Nov 03 22:44:51.411526 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.411502 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.430538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.426469 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb1fa5\"/\"openshift-service-ca.crt\"" Nov 03 22:44:51.430538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.426687 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb1fa5\"/\"default-dockercfg-22v64\"" Nov 03 22:44:51.430538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.426835 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bb1fa5\"/\"kube-root-ca.crt\"" Nov 03 22:44:51.430538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.427718 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bb1fa5\"/\"test-model-bb1fa5-kserve-self-signed-certs\"" Nov 03 22:44:51.441349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.439996 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw"] Nov 03 22:44:51.538971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.538929 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1527d9f-2f08-4e37-88ea-22bbca92705b-tls-certs\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.539164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.539037 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-home\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.539164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.539156 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-dshm\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.539329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.539200 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-tmp-dir\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.539412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.539341 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl5p4\" (UniqueName: \"kubernetes.io/projected/a1527d9f-2f08-4e37-88ea-22bbca92705b-kube-api-access-zl5p4\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.539509 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.539486 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-model-cache\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.641659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.641575 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl5p4\" (UniqueName: \"kubernetes.io/projected/a1527d9f-2f08-4e37-88ea-22bbca92705b-kube-api-access-zl5p4\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.641822 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.641675 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-model-cache\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.641894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.641838 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1527d9f-2f08-4e37-88ea-22bbca92705b-tls-certs\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.641894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.641871 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-home\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.642019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.641932 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-dshm\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.642019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.641977 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-tmp-dir\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.642159 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.642123 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-model-cache\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.642339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.642263 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-home\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.642403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.642392 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-tmp-dir\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.644160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.644139 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-dshm\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.644446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.644425 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1527d9f-2f08-4e37-88ea-22bbca92705b-tls-certs\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.663822 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.663795 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl5p4\" (UniqueName: \"kubernetes.io/projected/a1527d9f-2f08-4e37-88ea-22bbca92705b-kube-api-access-zl5p4\") pod \"test-model-bb1fa5-kserve-7d75bcf989-kxhdw\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:51.737103 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:51.737066 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:52.189329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.186812 4903 patch_prober.go:29] interesting pod/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:44:52.189329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.186863 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:44:52.235431 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.235390 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw"] Nov 03 22:44:52.387789 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.387749 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" event={"ID":"a1527d9f-2f08-4e37-88ea-22bbca92705b","Type":"ContainerStarted","Data":"9541d1475e6f98ceeb0a69f11a1f0957e30906d7792893415a5f374ae3b08d51"} Nov 03 22:44:52.599525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.599491 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:52.689817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.689776 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300945fb-de39-48f9-982f-566890adfb48-kube-api-access-m46ww\" (UniqueName: \"kubernetes.io/projected/300945fb-de39-48f9-982f-566890adfb48-kube-api-access-m46ww\") pod \"300945fb-de39-48f9-982f-566890adfb48\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " Nov 03 22:44:52.690002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.689888 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300945fb-de39-48f9-982f-566890adfb48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/300945fb-de39-48f9-982f-566890adfb48-maas-api-openshift-ca\") pod \"300945fb-de39-48f9-982f-566890adfb48\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " Nov 03 22:44:52.690002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.689980 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/300945fb-de39-48f9-982f-566890adfb48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/300945fb-de39-48f9-982f-566890adfb48-maas-api-xks-ca\") pod \"300945fb-de39-48f9-982f-566890adfb48\" (UID: \"300945fb-de39-48f9-982f-566890adfb48\") " Nov 03 22:44:52.690367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.690337 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300945fb-de39-48f9-982f-566890adfb48-maas-api-openshift-ca" pod "300945fb-de39-48f9-982f-566890adfb48" (UID: "300945fb-de39-48f9-982f-566890adfb48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:52.690850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.690828 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/300945fb-de39-48f9-982f-566890adfb48-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:52.692225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.692200 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300945fb-de39-48f9-982f-566890adfb48-kube-api-access-m46ww" pod "300945fb-de39-48f9-982f-566890adfb48" (UID: "300945fb-de39-48f9-982f-566890adfb48"). InnerVolumeSpecName "kube-api-access-m46ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:52.692495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.692477 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300945fb-de39-48f9-982f-566890adfb48-maas-api-xks-ca" pod "300945fb-de39-48f9-982f-566890adfb48" (UID: "300945fb-de39-48f9-982f-566890adfb48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:44:52.793548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.793472 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/300945fb-de39-48f9-982f-566890adfb48-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:52.793548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:52.793500 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m46ww\" (UniqueName: \"kubernetes.io/projected/300945fb-de39-48f9-982f-566890adfb48-kube-api-access-m46ww\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.393336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.393310 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:44:53.399731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.399690 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" event={"ID":"a1527d9f-2f08-4e37-88ea-22bbca92705b","Type":"ContainerStarted","Data":"1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba"} Nov 03 22:44:53.401485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.401458 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" event={"ID":"300945fb-de39-48f9-982f-566890adfb48","Type":"ContainerDied","Data":"49a924b58894e3fb457a4ac1d7652126f037dadd2c2eef1894dae9c3093acd0d"} Nov 03 22:44:53.401609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.401502 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a924b58894e3fb457a4ac1d7652126f037dadd2c2eef1894dae9c3093acd0d" Nov 03 22:44:53.401609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.401552 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp" Nov 03 22:44:53.403385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.403351 4903 generic.go:356] "Generic (PLEG): container finished" podID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerID="6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906" exitCode=0 Nov 03 22:44:53.403487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.403406 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" Nov 03 22:44:53.403487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.403438 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" event={"ID":"0239ba34-b4fd-44c7-9c54-c908dba5353e","Type":"ContainerDied","Data":"6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906"} Nov 03 22:44:53.403575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.403501 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq" event={"ID":"0239ba34-b4fd-44c7-9c54-c908dba5353e","Type":"ContainerDied","Data":"8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf"} Nov 03 22:44:53.403575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.403535 4903 scope.go:122] "RemoveContainer" containerID="6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906" Nov 03 22:44:53.420074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.420041 4903 scope.go:122] "RemoveContainer" containerID="6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906" Nov 03 22:44:53.420642 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:44:53.420505 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906\": container with ID starting with 6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906 not found: ID does not exist" containerID="6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906" Nov 03 22:44:53.420642 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.420548 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906"} err="failed to get container status \"6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906\": rpc error: code = NotFound desc = could not find container \"6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906\": container with ID starting with 6a0f66b14d9fcbe8ad4a6ff2a558268351be15e33df820af4a92696189231906 not found: ID does not exist" Nov 03 22:44:53.571318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.545819 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0239ba34-b4fd-44c7-9c54-c908dba5353e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0239ba34-b4fd-44c7-9c54-c908dba5353e-istiod-ca-cert\") pod \"0239ba34-b4fd-44c7-9c54-c908dba5353e\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " Nov 03 22:44:53.571318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.545949 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-data\") pod \"0239ba34-b4fd-44c7-9c54-c908dba5353e\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " Nov 03 22:44:53.571318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.546015 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-socket\") pod \"0239ba34-b4fd-44c7-9c54-c908dba5353e\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " Nov 03 22:44:53.571318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.546073 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-podinfo\") pod \"0239ba34-b4fd-44c7-9c54-c908dba5353e\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " Nov 03 22:44:53.571318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.546101 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-certs\") pod \"0239ba34-b4fd-44c7-9c54-c908dba5353e\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " Nov 03 22:44:53.571318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.546216 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-credential-socket\") pod \"0239ba34-b4fd-44c7-9c54-c908dba5353e\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " Nov 03 22:44:53.571318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.546498 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-envoy\") pod \"0239ba34-b4fd-44c7-9c54-c908dba5353e\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " Nov 03 22:44:53.571318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.546541 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-kube-api-access-mkq59\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-kube-api-access-mkq59\") pod \"0239ba34-b4fd-44c7-9c54-c908dba5353e\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " Nov 03 22:44:53.571318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.546596 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-token\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-token\") pod \"0239ba34-b4fd-44c7-9c54-c908dba5353e\" (UID: \"0239ba34-b4fd-44c7-9c54-c908dba5353e\") " Nov 03 22:44:53.593449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.593385 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-data" pod "0239ba34-b4fd-44c7-9c54-c908dba5353e" (UID: "0239ba34-b4fd-44c7-9c54-c908dba5353e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:53.617361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.598473 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-socket" pod "0239ba34-b4fd-44c7-9c54-c908dba5353e" (UID: "0239ba34-b4fd-44c7-9c54-c908dba5353e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:53.639329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.637769 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-credential-socket" pod "0239ba34-b4fd-44c7-9c54-c908dba5353e" (UID: "0239ba34-b4fd-44c7-9c54-c908dba5353e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:53.639505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.639440 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-certs" pod "0239ba34-b4fd-44c7-9c54-c908dba5353e" (UID: "0239ba34-b4fd-44c7-9c54-c908dba5353e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:53.643347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.640532 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0239ba34-b4fd-44c7-9c54-c908dba5353e-istiod-ca-cert" pod "0239ba34-b4fd-44c7-9c54-c908dba5353e" (UID: "0239ba34-b4fd-44c7-9c54-c908dba5353e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:44:53.688423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.674495 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0239ba34-b4fd-44c7-9c54-c908dba5353e-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.688423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.674526 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.688423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.674540 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.688423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.674553 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.688423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.674566 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.730708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.724474 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-token" pod "0239ba34-b4fd-44c7-9c54-c908dba5353e" (UID: "0239ba34-b4fd-44c7-9c54-c908dba5353e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:53.750354 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.749674 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" podStartSLOduration=2.749654385 podStartE2EDuration="2.749654385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:44:53.56872178 +0000 UTC m=+1919.972662755" watchObservedRunningTime="2026-11-03 22:44:53.749654385 +0000 UTC m=+1920.153595361" Nov 03 22:44:53.753133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.753093 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-kube-api-access-mkq59" pod "0239ba34-b4fd-44c7-9c54-c908dba5353e" (UID: "0239ba34-b4fd-44c7-9c54-c908dba5353e"). InnerVolumeSpecName "kube-api-access-mkq59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:44:53.800955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.777274 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkq59\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-kube-api-access-mkq59\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.800955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.777328 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.800955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.778050 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-envoy" pod "0239ba34-b4fd-44c7-9c54-c908dba5353e" (UID: "0239ba34-b4fd-44c7-9c54-c908dba5353e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:44:53.800955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.781389 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-podinfo" pod "0239ba34-b4fd-44c7-9c54-c908dba5353e" (UID: "0239ba34-b4fd-44c7-9c54-c908dba5353e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:44:53.859604 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.859573 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:44:53.879256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.879103 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.879256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:53.879180 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0239ba34-b4fd-44c7-9c54-c908dba5353e-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:44:53.974226 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:44:53.974189 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:44:54.050280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:54.050249 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq"] Nov 03 22:44:54.060417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:54.060381 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e19850-openshift-default-9f855bd47-q8mkq"] Nov 03 22:44:55.740160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:55.740117 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" path="/var/lib/kubelet/pods/0239ba34-b4fd-44c7-9c54-c908dba5353e/volumes" Nov 03 22:44:57.985861 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:44:57.985818 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:44:58.171866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:58.171833 4903 scope.go:122] "RemoveContainer" containerID="c238b9dd2e8c7c62029d99b833895eed2591f2869806e8b5b074c20c2de4c842" Nov 03 22:44:58.188096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:58.188074 4903 scope.go:122] "RemoveContainer" containerID="6ccf2f8664db98ff0516b3deca43d9077fbf27355e5a8da6daf60966e7f7695c" Nov 03 22:44:58.202286 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:58.202263 4903 scope.go:122] "RemoveContainer" containerID="49a84705010cfb4aa65ac2ce0b1e637cc8e8cfd0df46ccbb7732edc4a5174614" Nov 03 22:44:58.215159 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:44:58.215040 4903 scope.go:122] "RemoveContainer" containerID="51c0ea122a59378cd8a044db4daec8cb335ba11483e62036c47323118ee2b612" Nov 03 22:45:00.153285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.153218 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp"] Nov 03 22:45:00.154791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.154770 4903 cpu_manager.go:414] "containerMap: removing container" podUID="300945fb-de39-48f9-982f-566890adfb48" containerName="revoke-keys" Nov 03 22:45:00.154853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.154794 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300945fb-de39-48f9-982f-566890adfb48" containerName="revoke-keys" Nov 03 22:45:00.154853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.154814 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" Nov 03 22:45:00.154853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.154822 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" Nov 03 22:45:00.155112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.155098 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="300945fb-de39-48f9-982f-566890adfb48" containerName="revoke-keys" Nov 03 22:45:00.155168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.155119 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0239ba34-b4fd-44c7-9c54-c908dba5353e" containerName="istio-proxy" Nov 03 22:45:00.163398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.163364 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps"] Nov 03 22:45:00.165902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.165012 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:00.170483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.170460 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:00.170797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.170776 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5"] Nov 03 22:45:00.176336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.176270 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:00.178461 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.178441 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv"] Nov 03 22:45:00.188132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.188103 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:45:00.188278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.188185 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:45:00.188278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.188263 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:45:00.206655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.199646 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsf5\" (UniqueName: \"kubernetes.io/projected/85615752-db5a-4dea-bc86-290e793928e0-kube-api-access-jgsf5\") pod \"maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp\" (UID: \"85615752-db5a-4dea-bc86-290e793928e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:00.206655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.199697 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85615752-db5a-4dea-bc86-290e793928e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp\" (UID: \"85615752-db5a-4dea-bc86-290e793928e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:00.206655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.199758 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfv5\" (UniqueName: \"kubernetes.io/projected/3ec653a6-42f1-49b3-a7d9-827945ef8946-kube-api-access-fvfv5\") pod \"maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps\" (UID: \"3ec653a6-42f1-49b3-a7d9-827945ef8946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:00.206655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.199824 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec653a6-42f1-49b3-a7d9-827945ef8946-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps\" (UID: \"3ec653a6-42f1-49b3-a7d9-827945ef8946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:00.206655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.202511 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh"] Nov 03 22:45:00.206655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.203353 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:00.209453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.208605 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp"] Nov 03 22:45:00.209453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.208716 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:00.238390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.238353 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps"] Nov 03 22:45:00.244919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.244888 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5"] Nov 03 22:45:00.249528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.249498 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv"] Nov 03 22:45:00.256941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.256397 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh"] Nov 03 22:45:00.282351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.282324 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s"] Nov 03 22:45:00.287419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.287394 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:00.300410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.300378 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62"] Nov 03 22:45:00.302720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.302689 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7h9\" (UniqueName: \"kubernetes.io/projected/9445c683-e8c3-41bf-9cbc-5d3093d840c6-kube-api-access-5g7h9\") pod \"maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5\" (UID: \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:00.302862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.302744 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c71cca5-adfa-4f08-9233-2719cadc9550-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh\" (UID: \"7c71cca5-adfa-4f08-9233-2719cadc9550\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:00.302862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.302808 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwnpb\" (UniqueName: \"kubernetes.io/projected/7c71cca5-adfa-4f08-9233-2719cadc9550-kube-api-access-nwnpb\") pod \"maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh\" (UID: \"7c71cca5-adfa-4f08-9233-2719cadc9550\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:00.302944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.302887 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsf5\" (UniqueName: \"kubernetes.io/projected/85615752-db5a-4dea-bc86-290e793928e0-kube-api-access-jgsf5\") pod \"maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp\" (UID: \"85615752-db5a-4dea-bc86-290e793928e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:00.302944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.302927 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85615752-db5a-4dea-bc86-290e793928e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp\" (UID: \"85615752-db5a-4dea-bc86-290e793928e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:00.303020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.302983 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445c683-e8c3-41bf-9cbc-5d3093d840c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5\" (UID: \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:00.303059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.303017 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfv5\" (UniqueName: \"kubernetes.io/projected/3ec653a6-42f1-49b3-a7d9-827945ef8946-kube-api-access-fvfv5\") pod \"maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps\" (UID: \"3ec653a6-42f1-49b3-a7d9-827945ef8946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:00.303096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.303086 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec653a6-42f1-49b3-a7d9-827945ef8946-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps\" (UID: \"3ec653a6-42f1-49b3-a7d9-827945ef8946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:00.303418 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.303399 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqpqv\" (UniqueName: \"kubernetes.io/projected/1010f884-c3b1-442b-bd6d-848666e3bfb4-kube-api-access-gqpqv\") pod \"maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv\" (UID: \"1010f884-c3b1-442b-bd6d-848666e3bfb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:00.303463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.303442 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f884-c3b1-442b-bd6d-848666e3bfb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv\" (UID: \"1010f884-c3b1-442b-bd6d-848666e3bfb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:00.304285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.304262 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85615752-db5a-4dea-bc86-290e793928e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp\" (UID: \"85615752-db5a-4dea-bc86-290e793928e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:00.304530 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.304508 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec653a6-42f1-49b3-a7d9-827945ef8946-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps\" (UID: \"3ec653a6-42f1-49b3-a7d9-827945ef8946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:00.307395 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.307367 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw"] Nov 03 22:45:00.307515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.307477 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:00.313628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.313547 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq"] Nov 03 22:45:00.313934 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.313689 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:00.345358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.323510 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p"] Nov 03 22:45:00.345358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.324814 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:00.345358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.344378 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsf5\" (UniqueName: \"kubernetes.io/projected/85615752-db5a-4dea-bc86-290e793928e0-kube-api-access-jgsf5\") pod \"maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp\" (UID: \"85615752-db5a-4dea-bc86-290e793928e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:00.357250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.354868 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfv5\" (UniqueName: \"kubernetes.io/projected/3ec653a6-42f1-49b3-a7d9-827945ef8946-kube-api-access-fvfv5\") pod \"maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps\" (UID: \"3ec653a6-42f1-49b3-a7d9-827945ef8946\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:00.357250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.355391 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s"] Nov 03 22:45:00.357250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.355459 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw"] Nov 03 22:45:00.357250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.355488 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq"] Nov 03 22:45:00.357250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.355571 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:00.359538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.359515 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62"] Nov 03 22:45:00.379576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.376012 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj"] Nov 03 22:45:00.411619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.397643 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg"] Nov 03 22:45:00.412322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.412055 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:00.427584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.427364 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2frhz\" (UniqueName: \"kubernetes.io/projected/4e3fdb74-d353-4007-a96b-3af3db01cc57-kube-api-access-2frhz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62\" (UID: \"4e3fdb74-d353-4007-a96b-3af3db01cc57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:00.427584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.427515 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw\" (UID: \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:00.427831 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.427777 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqpqv\" (UniqueName: \"kubernetes.io/projected/1010f884-c3b1-442b-bd6d-848666e3bfb4-kube-api-access-gqpqv\") pod \"maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv\" (UID: \"1010f884-c3b1-442b-bd6d-848666e3bfb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:00.427897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.427833 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659a4a6e-2d77-4d9f-8284-a29be2d2de56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s\" (UID: \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:00.427897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.427869 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f884-c3b1-442b-bd6d-848666e3bfb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv\" (UID: \"1010f884-c3b1-442b-bd6d-848666e3bfb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:00.428046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428026 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7h9\" (UniqueName: \"kubernetes.io/projected/9445c683-e8c3-41bf-9cbc-5d3093d840c6-kube-api-access-5g7h9\") pod \"maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5\" (UID: \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:00.428113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428067 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrhp\" (UniqueName: \"kubernetes.io/projected/659a4a6e-2d77-4d9f-8284-a29be2d2de56-kube-api-access-rwrhp\") pod \"maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s\" (UID: \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:00.428113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428093 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c71cca5-adfa-4f08-9233-2719cadc9550-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh\" (UID: \"7c71cca5-adfa-4f08-9233-2719cadc9550\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:00.429025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428396 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66h6\" (UniqueName: \"kubernetes.io/projected/1b686411-6854-4f2a-9400-8632d8e55009-kube-api-access-g66h6\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq\" (UID: \"1b686411-6854-4f2a-9400-8632d8e55009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:00.429025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428465 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwnpb\" (UniqueName: \"kubernetes.io/projected/7c71cca5-adfa-4f08-9233-2719cadc9550-kube-api-access-nwnpb\") pod \"maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh\" (UID: \"7c71cca5-adfa-4f08-9233-2719cadc9550\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:00.429025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428500 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3fdb74-d353-4007-a96b-3af3db01cc57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62\" (UID: \"4e3fdb74-d353-4007-a96b-3af3db01cc57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:00.429025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428770 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb962f11-d246-41e8-a596-6383a2d91a58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p\" (UID: \"fb962f11-d246-41e8-a596-6383a2d91a58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:00.429025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428844 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b686411-6854-4f2a-9400-8632d8e55009-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq\" (UID: \"1b686411-6854-4f2a-9400-8632d8e55009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:00.429025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428940 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnj6\" (UniqueName: \"kubernetes.io/projected/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-kube-api-access-7lnj6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw\" (UID: \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:00.429025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.428946 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c71cca5-adfa-4f08-9233-2719cadc9550-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh\" (UID: \"7c71cca5-adfa-4f08-9233-2719cadc9550\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:00.429025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.429028 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thns8\" (UniqueName: \"kubernetes.io/projected/fb962f11-d246-41e8-a596-6383a2d91a58-kube-api-access-thns8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p\" (UID: \"fb962f11-d246-41e8-a596-6383a2d91a58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:00.429625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.429063 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445c683-e8c3-41bf-9cbc-5d3093d840c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5\" (UID: \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:00.429625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.429576 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f884-c3b1-442b-bd6d-848666e3bfb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv\" (UID: \"1010f884-c3b1-442b-bd6d-848666e3bfb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:00.429849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.429827 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445c683-e8c3-41bf-9cbc-5d3093d840c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5\" (UID: \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:00.443695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.442504 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p"] Nov 03 22:45:00.443695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.442601 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:00.450638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.450614 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p"] Nov 03 22:45:00.450638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.450661 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r"] Nov 03 22:45:00.454191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.452069 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:00.458445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.456255 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj"] Nov 03 22:45:00.458445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.456387 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:00.463886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.463840 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg"] Nov 03 22:45:00.479974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.479951 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p"] Nov 03 22:45:00.485136 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.484833 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqpqv\" (UniqueName: \"kubernetes.io/projected/1010f884-c3b1-442b-bd6d-848666e3bfb4-kube-api-access-gqpqv\") pod \"maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv\" (UID: \"1010f884-c3b1-442b-bd6d-848666e3bfb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:00.485136 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.484927 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:00.487701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.487675 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d"] Nov 03 22:45:00.492677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.488637 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwnpb\" (UniqueName: \"kubernetes.io/projected/7c71cca5-adfa-4f08-9233-2719cadc9550-kube-api-access-nwnpb\") pod \"maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh\" (UID: \"7c71cca5-adfa-4f08-9233-2719cadc9550\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:00.492677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.490535 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7h9\" (UniqueName: \"kubernetes.io/projected/9445c683-e8c3-41bf-9cbc-5d3093d840c6-kube-api-access-5g7h9\") pod \"maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5\" (UID: \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:00.497031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.496883 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:00.501684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.501660 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r"] Nov 03 22:45:00.512381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.512146 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:00.513853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.513598 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr"] Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531201 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rf6f\" (UniqueName: \"kubernetes.io/projected/31ad238a-db65-4160-a25b-9e29df9dbc94-kube-api-access-6rf6f\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r\" (UID: \"31ad238a-db65-4160-a25b-9e29df9dbc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531273 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb962f11-d246-41e8-a596-6383a2d91a58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p\" (UID: \"fb962f11-d246-41e8-a596-6383a2d91a58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531336 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b686411-6854-4f2a-9400-8632d8e55009-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq\" (UID: \"1b686411-6854-4f2a-9400-8632d8e55009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531375 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnj6\" (UniqueName: \"kubernetes.io/projected/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-kube-api-access-7lnj6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw\" (UID: \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531409 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjgh\" (UniqueName: \"kubernetes.io/projected/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-kube-api-access-pqjgh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p\" (UID: \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531444 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thns8\" (UniqueName: \"kubernetes.io/projected/fb962f11-d246-41e8-a596-6383a2d91a58-kube-api-access-thns8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p\" (UID: \"fb962f11-d246-41e8-a596-6383a2d91a58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531535 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7sw5\" (UniqueName: \"kubernetes.io/projected/663b664e-98af-4583-aed0-da9572e83f52-kube-api-access-g7sw5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj\" (UID: \"663b664e-98af-4583-aed0-da9572e83f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531579 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p\" (UID: \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531641 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2frhz\" (UniqueName: \"kubernetes.io/projected/4e3fdb74-d353-4007-a96b-3af3db01cc57-kube-api-access-2frhz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62\" (UID: \"4e3fdb74-d353-4007-a96b-3af3db01cc57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531696 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whh9n\" (UniqueName: \"kubernetes.io/projected/c9d6c159-15e1-4fc7-8d50-40758bd126be-kube-api-access-whh9n\") pod \"maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg\" (UID: \"c9d6c159-15e1-4fc7-8d50-40758bd126be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531740 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw\" (UID: \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531899 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b664e-98af-4583-aed0-da9572e83f52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj\" (UID: \"663b664e-98af-4583-aed0-da9572e83f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.531954 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659a4a6e-2d77-4d9f-8284-a29be2d2de56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s\" (UID: \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.532034 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrhp\" (UniqueName: \"kubernetes.io/projected/659a4a6e-2d77-4d9f-8284-a29be2d2de56-kube-api-access-rwrhp\") pod \"maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s\" (UID: \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.532075 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ad238a-db65-4160-a25b-9e29df9dbc94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r\" (UID: \"31ad238a-db65-4160-a25b-9e29df9dbc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.532128 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66h6\" (UniqueName: \"kubernetes.io/projected/1b686411-6854-4f2a-9400-8632d8e55009-kube-api-access-g66h6\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq\" (UID: \"1b686411-6854-4f2a-9400-8632d8e55009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.532169 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3fdb74-d353-4007-a96b-3af3db01cc57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62\" (UID: \"4e3fdb74-d353-4007-a96b-3af3db01cc57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.532231 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6c159-15e1-4fc7-8d50-40758bd126be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg\" (UID: \"c9d6c159-15e1-4fc7-8d50-40758bd126be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.533051 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb962f11-d246-41e8-a596-6383a2d91a58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p\" (UID: \"fb962f11-d246-41e8-a596-6383a2d91a58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:00.534049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.533693 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b686411-6854-4f2a-9400-8632d8e55009-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq\" (UID: \"1b686411-6854-4f2a-9400-8632d8e55009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:00.538252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.535277 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw\" (UID: \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:00.538252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.535909 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659a4a6e-2d77-4d9f-8284-a29be2d2de56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s\" (UID: \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:00.538252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.536907 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3fdb74-d353-4007-a96b-3af3db01cc57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62\" (UID: \"4e3fdb74-d353-4007-a96b-3af3db01cc57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:00.538252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.537112 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:00.584747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.556587 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:00.588398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.588368 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnj6\" (UniqueName: \"kubernetes.io/projected/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-kube-api-access-7lnj6\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw\" (UID: \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:00.624783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.595600 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66h6\" (UniqueName: \"kubernetes.io/projected/1b686411-6854-4f2a-9400-8632d8e55009-kube-api-access-g66h6\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq\" (UID: \"1b686411-6854-4f2a-9400-8632d8e55009\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:00.624783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.598818 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrhp\" (UniqueName: \"kubernetes.io/projected/659a4a6e-2d77-4d9f-8284-a29be2d2de56-kube-api-access-rwrhp\") pod \"maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s\" (UID: \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:00.624783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.610832 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:00.624783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.619402 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653183 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7sw5\" (UniqueName: \"kubernetes.io/projected/663b664e-98af-4583-aed0-da9572e83f52-kube-api-access-g7sw5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj\" (UID: \"663b664e-98af-4583-aed0-da9572e83f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653243 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p\" (UID: \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653346 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whh9n\" (UniqueName: \"kubernetes.io/projected/c9d6c159-15e1-4fc7-8d50-40758bd126be-kube-api-access-whh9n\") pod \"maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg\" (UID: \"c9d6c159-15e1-4fc7-8d50-40758bd126be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653505 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b664e-98af-4583-aed0-da9572e83f52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj\" (UID: \"663b664e-98af-4583-aed0-da9572e83f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653601 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ad238a-db65-4160-a25b-9e29df9dbc94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r\" (UID: \"31ad238a-db65-4160-a25b-9e29df9dbc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653684 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6c159-15e1-4fc7-8d50-40758bd126be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg\" (UID: \"c9d6c159-15e1-4fc7-8d50-40758bd126be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653716 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rf6f\" (UniqueName: \"kubernetes.io/projected/31ad238a-db65-4160-a25b-9e29df9dbc94-kube-api-access-6rf6f\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r\" (UID: \"31ad238a-db65-4160-a25b-9e29df9dbc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653761 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598378e1-2604-411f-9b3c-e2a196b4dc2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d\" (UID: \"598378e1-2604-411f-9b3c-e2a196b4dc2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653821 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjgh\" (UniqueName: \"kubernetes.io/projected/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-kube-api-access-pqjgh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p\" (UID: \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.653878 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxzz2\" (UniqueName: \"kubernetes.io/projected/598378e1-2604-411f-9b3c-e2a196b4dc2e-kube-api-access-sxzz2\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d\" (UID: \"598378e1-2604-411f-9b3c-e2a196b4dc2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.654449 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p\" (UID: \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:00.663622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.654941 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b664e-98af-4583-aed0-da9572e83f52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj\" (UID: \"663b664e-98af-4583-aed0-da9572e83f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:00.664720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.664038 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ad238a-db65-4160-a25b-9e29df9dbc94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r\" (UID: \"31ad238a-db65-4160-a25b-9e29df9dbc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:00.664720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.664528 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6c159-15e1-4fc7-8d50-40758bd126be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg\" (UID: \"c9d6c159-15e1-4fc7-8d50-40758bd126be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:00.671351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.671280 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:00.676401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.676232 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whh9n\" (UniqueName: \"kubernetes.io/projected/c9d6c159-15e1-4fc7-8d50-40758bd126be-kube-api-access-whh9n\") pod \"maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg\" (UID: \"c9d6c159-15e1-4fc7-8d50-40758bd126be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:00.681535 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.681150 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjgh\" (UniqueName: \"kubernetes.io/projected/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-kube-api-access-pqjgh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p\" (UID: \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:00.682509 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.681705 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d"] Nov 03 22:45:00.688974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.688951 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc"] Nov 03 22:45:00.690660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.690413 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:00.693656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.691481 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thns8\" (UniqueName: \"kubernetes.io/projected/fb962f11-d246-41e8-a596-6383a2d91a58-kube-api-access-thns8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p\" (UID: \"fb962f11-d246-41e8-a596-6383a2d91a58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:00.693656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.693533 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rf6f\" (UniqueName: \"kubernetes.io/projected/31ad238a-db65-4160-a25b-9e29df9dbc94-kube-api-access-6rf6f\") pod \"maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r\" (UID: \"31ad238a-db65-4160-a25b-9e29df9dbc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:00.699771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.695798 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2frhz\" (UniqueName: \"kubernetes.io/projected/4e3fdb74-d353-4007-a96b-3af3db01cc57-kube-api-access-2frhz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62\" (UID: \"4e3fdb74-d353-4007-a96b-3af3db01cc57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:00.699771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.696495 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7sw5\" (UniqueName: \"kubernetes.io/projected/663b664e-98af-4583-aed0-da9572e83f52-kube-api-access-g7sw5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj\" (UID: \"663b664e-98af-4583-aed0-da9572e83f52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:00.699771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.697950 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn"] Nov 03 22:45:00.699771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.699529 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:00.718205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.718070 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:00.718400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.718383 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr"] Nov 03 22:45:00.720499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.720422 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:00.728318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.726677 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:00.731920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.731900 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:00.743579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.743067 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc"] Nov 03 22:45:00.755041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.754903 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9725d5-d50c-496b-834e-a783e0c47b28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc\" (UID: \"1d9725d5-d50c-496b-834e-a783e0c47b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:00.755041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.754967 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljk2w\" (UniqueName: \"kubernetes.io/projected/1d9725d5-d50c-496b-834e-a783e0c47b28-kube-api-access-ljk2w\") pod \"maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc\" (UID: \"1d9725d5-d50c-496b-834e-a783e0c47b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:00.756011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.755489 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598378e1-2604-411f-9b3c-e2a196b4dc2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d\" (UID: \"598378e1-2604-411f-9b3c-e2a196b4dc2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:00.756011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.755578 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxzz2\" (UniqueName: \"kubernetes.io/projected/598378e1-2604-411f-9b3c-e2a196b4dc2e-kube-api-access-sxzz2\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d\" (UID: \"598378e1-2604-411f-9b3c-e2a196b4dc2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:00.756011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.755691 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mhc\" (UniqueName: \"kubernetes.io/projected/b10ff0e7-d241-428a-9e02-ac902f55b8c1-kube-api-access-s5mhc\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr\" (UID: \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:00.756011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.755812 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ff0e7-d241-428a-9e02-ac902f55b8c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr\" (UID: \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:00.756846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.756800 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598378e1-2604-411f-9b3c-e2a196b4dc2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d\" (UID: \"598378e1-2604-411f-9b3c-e2a196b4dc2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:00.756954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.756879 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:00.781437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.778225 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:00.786922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.786714 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:00.798950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.794553 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q"] Nov 03 22:45:00.807705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.807670 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxzz2\" (UniqueName: \"kubernetes.io/projected/598378e1-2604-411f-9b3c-e2a196b4dc2e-kube-api-access-sxzz2\") pod \"maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d\" (UID: \"598378e1-2604-411f-9b3c-e2a196b4dc2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:00.814366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.812920 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4"] Nov 03 22:45:00.815870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.815388 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:00.828546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.828513 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn"] Nov 03 22:45:00.828704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.828690 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:00.837610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.837454 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q"] Nov 03 22:45:00.864179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.860411 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9725d5-d50c-496b-834e-a783e0c47b28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc\" (UID: \"1d9725d5-d50c-496b-834e-a783e0c47b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:00.864179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.860483 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljk2w\" (UniqueName: \"kubernetes.io/projected/1d9725d5-d50c-496b-834e-a783e0c47b28-kube-api-access-ljk2w\") pod \"maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc\" (UID: \"1d9725d5-d50c-496b-834e-a783e0c47b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:00.864179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.860580 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f271965-1d01-4f06-8619-ac80aaeb2154-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn\" (UID: \"2f271965-1d01-4f06-8619-ac80aaeb2154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:00.864179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.860825 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mhc\" (UniqueName: \"kubernetes.io/projected/b10ff0e7-d241-428a-9e02-ac902f55b8c1-kube-api-access-s5mhc\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr\" (UID: \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:00.864179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.860978 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ff0e7-d241-428a-9e02-ac902f55b8c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr\" (UID: \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:00.864179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.861032 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gvt8\" (UniqueName: \"kubernetes.io/projected/2f271965-1d01-4f06-8619-ac80aaeb2154-kube-api-access-7gvt8\") pod \"maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn\" (UID: \"2f271965-1d01-4f06-8619-ac80aaeb2154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:00.864179 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.862586 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4"] Nov 03 22:45:00.865130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.865059 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ff0e7-d241-428a-9e02-ac902f55b8c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr\" (UID: \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:00.865130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.865111 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9725d5-d50c-496b-834e-a783e0c47b28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc\" (UID: \"1d9725d5-d50c-496b-834e-a783e0c47b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:00.923571 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.922130 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljk2w\" (UniqueName: \"kubernetes.io/projected/1d9725d5-d50c-496b-834e-a783e0c47b28-kube-api-access-ljk2w\") pod \"maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc\" (UID: \"1d9725d5-d50c-496b-834e-a783e0c47b28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:00.925489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.925302 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mhc\" (UniqueName: \"kubernetes.io/projected/b10ff0e7-d241-428a-9e02-ac902f55b8c1-kube-api-access-s5mhc\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr\" (UID: \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:00.956170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.955982 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:00.969397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.968065 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrqsq\" (UniqueName: \"kubernetes.io/projected/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-kube-api-access-vrqsq\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q\" (UID: \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:00.969397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.968231 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f271965-1d01-4f06-8619-ac80aaeb2154-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn\" (UID: \"2f271965-1d01-4f06-8619-ac80aaeb2154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:00.969397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.968347 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lx7g\" (UniqueName: \"kubernetes.io/projected/b052a69f-e1a3-408e-9881-94770cb92ed6-kube-api-access-9lx7g\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4\" (UID: \"b052a69f-e1a3-408e-9881-94770cb92ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:00.969397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.968495 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q\" (UID: \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:00.969397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.968543 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gvt8\" (UniqueName: \"kubernetes.io/projected/2f271965-1d01-4f06-8619-ac80aaeb2154-kube-api-access-7gvt8\") pod \"maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn\" (UID: \"2f271965-1d01-4f06-8619-ac80aaeb2154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:00.969397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:00.968622 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b052a69f-e1a3-408e-9881-94770cb92ed6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4\" (UID: \"b052a69f-e1a3-408e-9881-94770cb92ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:01.014145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.013962 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f271965-1d01-4f06-8619-ac80aaeb2154-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn\" (UID: \"2f271965-1d01-4f06-8619-ac80aaeb2154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:01.050666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.050641 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gvt8\" (UniqueName: \"kubernetes.io/projected/2f271965-1d01-4f06-8619-ac80aaeb2154-kube-api-access-7gvt8\") pod \"maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn\" (UID: \"2f271965-1d01-4f06-8619-ac80aaeb2154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:01.052012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.051787 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:01.071433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.071390 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lx7g\" (UniqueName: \"kubernetes.io/projected/b052a69f-e1a3-408e-9881-94770cb92ed6-kube-api-access-9lx7g\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4\" (UID: \"b052a69f-e1a3-408e-9881-94770cb92ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:01.071635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.071600 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q\" (UID: \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:01.071762 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.071744 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b052a69f-e1a3-408e-9881-94770cb92ed6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4\" (UID: \"b052a69f-e1a3-408e-9881-94770cb92ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:01.071871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.071855 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrqsq\" (UniqueName: \"kubernetes.io/projected/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-kube-api-access-vrqsq\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q\" (UID: \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:01.075606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.075369 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b052a69f-e1a3-408e-9881-94770cb92ed6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4\" (UID: \"b052a69f-e1a3-408e-9881-94770cb92ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:01.077861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.077074 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q\" (UID: \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:01.089279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.089226 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:01.102540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.102474 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lx7g\" (UniqueName: \"kubernetes.io/projected/b052a69f-e1a3-408e-9881-94770cb92ed6-kube-api-access-9lx7g\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4\" (UID: \"b052a69f-e1a3-408e-9881-94770cb92ed6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:01.116399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.116148 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:01.139581 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.138654 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrqsq\" (UniqueName: \"kubernetes.io/projected/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-kube-api-access-vrqsq\") pod \"maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q\" (UID: \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:01.170749 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.170512 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:01.208820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.208552 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:01.387279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.386896 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:01.423514 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.423474 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp"] Nov 03 22:45:01.443694 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:01.443662 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85615752_db5a_4dea_bc86_290e793928e0.slice/crio-48c07598dadf2f28ebe4ee2875e0734db98c64fffff4e89ca0994033a92d8454 WatchSource:0}: Error finding container 48c07598dadf2f28ebe4ee2875e0734db98c64fffff4e89ca0994033a92d8454: Status 404 returned error can't find the container with id 48c07598dadf2f28ebe4ee2875e0734db98c64fffff4e89ca0994033a92d8454 Nov 03 22:45:01.708173 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:01.708132 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec653a6_42f1_49b3_a7d9_827945ef8946.slice/crio-6a7c73a93ede72590d76aecccbdbb22f3c9c51794305868605348071f6fce804 WatchSource:0}: Error finding container 6a7c73a93ede72590d76aecccbdbb22f3c9c51794305868605348071f6fce804: Status 404 returned error can't find the container with id 6a7c73a93ede72590d76aecccbdbb22f3c9c51794305868605348071f6fce804 Nov 03 22:45:01.723395 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.710149 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps"] Nov 03 22:45:01.723635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.723610 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" event={"ID":"85615752-db5a-4dea-bc86-290e793928e0","Type":"ContainerStarted","Data":"48c07598dadf2f28ebe4ee2875e0734db98c64fffff4e89ca0994033a92d8454"} Nov 03 22:45:01.759791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.754514 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5"] Nov 03 22:45:01.759791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.754595 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:45:01.781721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.781691 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:45:01.997950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:01.997865 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn"] Nov 03 22:45:02.003826 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:02.003799 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f271965_1d01_4f06_8619_ac80aaeb2154.slice/crio-e843963bee2d77c83e8df5023361362c5daad0259e3b3efc18f3c5511f7f407d WatchSource:0}: Error finding container e843963bee2d77c83e8df5023361362c5daad0259e3b3efc18f3c5511f7f407d: Status 404 returned error can't find the container with id e843963bee2d77c83e8df5023361362c5daad0259e3b3efc18f3c5511f7f407d Nov 03 22:45:02.005523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.005463 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh"] Nov 03 22:45:02.006465 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:02.006444 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c71cca5_adfa_4f08_9233_2719cadc9550.slice/crio-fbddc9beb04d09ce0fa260b7b26cae8f36f60e2c71a4d6e979c4c1edae6a08f6 WatchSource:0}: Error finding container fbddc9beb04d09ce0fa260b7b26cae8f36f60e2c71a4d6e979c4c1edae6a08f6: Status 404 returned error can't find the container with id fbddc9beb04d09ce0fa260b7b26cae8f36f60e2c71a4d6e979c4c1edae6a08f6 Nov 03 22:45:02.009686 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:02.009663 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63f0f29_03cf_4880_8e34_b44f25a1eb2a.slice/crio-03fc769d8cfb099f6c5c6cf70520caccc8867324be7d41402f6195449826e485 WatchSource:0}: Error finding container 03fc769d8cfb099f6c5c6cf70520caccc8867324be7d41402f6195449826e485: Status 404 returned error can't find the container with id 03fc769d8cfb099f6c5c6cf70520caccc8867324be7d41402f6195449826e485 Nov 03 22:45:02.012947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.012924 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p"] Nov 03 22:45:02.021694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.021660 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q"] Nov 03 22:45:02.022763 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:02.022740 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be9a4b9_2039_4d10_b1a5_f6a3cd3b2164.slice/crio-8049f64151c407c6fa9f090c4bf348fa6c5989e49fbd3a57f660c64355452ca0 WatchSource:0}: Error finding container 8049f64151c407c6fa9f090c4bf348fa6c5989e49fbd3a57f660c64355452ca0: Status 404 returned error can't find the container with id 8049f64151c407c6fa9f090c4bf348fa6c5989e49fbd3a57f660c64355452ca0 Nov 03 22:45:02.048458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.048415 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc"] Nov 03 22:45:02.066142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.066059 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv"] Nov 03 22:45:02.073654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.073621 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s"] Nov 03 22:45:02.081735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.081712 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r"] Nov 03 22:45:02.100726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.100697 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj"] Nov 03 22:45:02.110219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.110190 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p"] Nov 03 22:45:02.117467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.116209 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr"] Nov 03 22:45:02.119266 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:02.119215 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb962f11_d246_41e8_a596_6383a2d91a58.slice/crio-789cad193ebcd275424bb5cef0802f4110f12443829adafd232ff9c0ca39e1b6 WatchSource:0}: Error finding container 789cad193ebcd275424bb5cef0802f4110f12443829adafd232ff9c0ca39e1b6: Status 404 returned error can't find the container with id 789cad193ebcd275424bb5cef0802f4110f12443829adafd232ff9c0ca39e1b6 Nov 03 22:45:02.120586 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:02.120351 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10ff0e7_d241_428a_9e02_ac902f55b8c1.slice/crio-3a5aa15fc7cc861a75400318353c39a900121e2812cb8948c1cacb0e31399de1 WatchSource:0}: Error finding container 3a5aa15fc7cc861a75400318353c39a900121e2812cb8948c1cacb0e31399de1: Status 404 returned error can't find the container with id 3a5aa15fc7cc861a75400318353c39a900121e2812cb8948c1cacb0e31399de1 Nov 03 22:45:02.159885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.159857 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d"] Nov 03 22:45:02.160280 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:02.160244 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d6c159_15e1_4fc7_8d50_40758bd126be.slice/crio-1f22291cc12bf648fc1009594b1bd7dff127901f80b1b5ddbf5532faa7c84f92 WatchSource:0}: Error finding container 1f22291cc12bf648fc1009594b1bd7dff127901f80b1b5ddbf5532faa7c84f92: Status 404 returned error can't find the container with id 1f22291cc12bf648fc1009594b1bd7dff127901f80b1b5ddbf5532faa7c84f92 Nov 03 22:45:02.169401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.169371 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg"] Nov 03 22:45:02.183517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.183493 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw"] Nov 03 22:45:02.190878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.188391 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4"] Nov 03 22:45:02.190878 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:02.189880 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd177e30a_9ef2_4d4d_a2be_3872a6bc9411.slice/crio-0bbe19079a42761847aaeb47ee364f73206a958e39b0e372aa9ada06f863ab2d WatchSource:0}: Error finding container 0bbe19079a42761847aaeb47ee364f73206a958e39b0e372aa9ada06f863ab2d: Status 404 returned error can't find the container with id 0bbe19079a42761847aaeb47ee364f73206a958e39b0e372aa9ada06f863ab2d Nov 03 22:45:02.195400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.195374 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62"] Nov 03 22:45:02.202501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.202473 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq"] Nov 03 22:45:02.744739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.744702 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" event={"ID":"1010f884-c3b1-442b-bd6d-848666e3bfb4","Type":"ContainerStarted","Data":"ac05d9f0706a8766fd8b751f4540e25661289c7e9917cc2b1ec079824afa0c55"} Nov 03 22:45:02.751634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.751594 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" event={"ID":"fb962f11-d246-41e8-a596-6383a2d91a58","Type":"ContainerStarted","Data":"789cad193ebcd275424bb5cef0802f4110f12443829adafd232ff9c0ca39e1b6"} Nov 03 22:45:02.753573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.753537 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" event={"ID":"598378e1-2604-411f-9b3c-e2a196b4dc2e","Type":"ContainerStarted","Data":"35bdda3520298e286ef4fb95ea082cba113be3a6bcb1b52a8e49477494a7a28b"} Nov 03 22:45:02.755002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.754979 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" event={"ID":"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164","Type":"ContainerStarted","Data":"8049f64151c407c6fa9f090c4bf348fa6c5989e49fbd3a57f660c64355452ca0"} Nov 03 22:45:02.756520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.756497 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" event={"ID":"9445c683-e8c3-41bf-9cbc-5d3093d840c6","Type":"ContainerStarted","Data":"143d19cf9fcf399143f23ca54a26ad14a0fbe9d08864dae427c91c55fc427b40"} Nov 03 22:45:02.756623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.756541 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" event={"ID":"9445c683-e8c3-41bf-9cbc-5d3093d840c6","Type":"ContainerStarted","Data":"6934d936ad0acd825bf6cdc6c998188244e34754c19ccafdc8df37df8455d376"} Nov 03 22:45:02.759128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.759105 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" event={"ID":"3ec653a6-42f1-49b3-a7d9-827945ef8946","Type":"ContainerStarted","Data":"df340e92253953bf9fa6a63be87e4d2e8a130551bb62ee62fe5abbc6688fc4a1"} Nov 03 22:45:02.759219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.759149 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" event={"ID":"3ec653a6-42f1-49b3-a7d9-827945ef8946","Type":"ContainerStarted","Data":"6a7c73a93ede72590d76aecccbdbb22f3c9c51794305868605348071f6fce804"} Nov 03 22:45:02.762234 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.762204 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" event={"ID":"2f271965-1d01-4f06-8619-ac80aaeb2154","Type":"ContainerStarted","Data":"e843963bee2d77c83e8df5023361362c5daad0259e3b3efc18f3c5511f7f407d"} Nov 03 22:45:02.764505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.764482 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:45:02.765240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.765214 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" event={"ID":"1b686411-6854-4f2a-9400-8632d8e55009","Type":"ContainerStarted","Data":"db475a8a533fb12d85049512e6e887b9f76a54e90dde26934c6484db86a2ffc3"} Nov 03 22:45:02.767097 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.767078 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" event={"ID":"31ad238a-db65-4160-a25b-9e29df9dbc94","Type":"ContainerStarted","Data":"17d5d8ff298769bf97b8615028d3ff5c3f8cf669769fb060cd40fe1d2940906d"} Nov 03 22:45:02.769015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.768995 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" event={"ID":"4e3fdb74-d353-4007-a96b-3af3db01cc57","Type":"ContainerStarted","Data":"e85b14f3519d78928d72225c1617f1c4a4487ffccd41aa41f7e9a58257d090bf"} Nov 03 22:45:02.770186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.770166 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" event={"ID":"c9d6c159-15e1-4fc7-8d50-40758bd126be","Type":"ContainerStarted","Data":"1f22291cc12bf648fc1009594b1bd7dff127901f80b1b5ddbf5532faa7c84f92"} Nov 03 22:45:02.771643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.771621 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" event={"ID":"1d9725d5-d50c-496b-834e-a783e0c47b28","Type":"ContainerStarted","Data":"ea13a1e0d941a86fa48eadb648e6fcc7affbcdc55a77edfa3a59320f9be8a3be"} Nov 03 22:45:02.772796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.772779 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" event={"ID":"c63f0f29-03cf-4880-8e34-b44f25a1eb2a","Type":"ContainerStarted","Data":"03fc769d8cfb099f6c5c6cf70520caccc8867324be7d41402f6195449826e485"} Nov 03 22:45:02.773934 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.773912 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" event={"ID":"b10ff0e7-d241-428a-9e02-ac902f55b8c1","Type":"ContainerStarted","Data":"3a5aa15fc7cc861a75400318353c39a900121e2812cb8948c1cacb0e31399de1"} Nov 03 22:45:02.775654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.775634 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" event={"ID":"663b664e-98af-4583-aed0-da9572e83f52","Type":"ContainerStarted","Data":"b6c2d65d8f9e41ad45aa312ee64941d6c47d7cecf892ac33dd4e8d58802ed84e"} Nov 03 22:45:02.776558 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.776542 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" event={"ID":"659a4a6e-2d77-4d9f-8284-a29be2d2de56","Type":"ContainerStarted","Data":"358a1eecd4878a65d0e1a025a4fd691885b8e4765e4fd38a80452fb758e89d2c"} Nov 03 22:45:02.777409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.777393 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" event={"ID":"d177e30a-9ef2-4d4d-a2be-3872a6bc9411","Type":"ContainerStarted","Data":"0bbe19079a42761847aaeb47ee364f73206a958e39b0e372aa9ada06f863ab2d"} Nov 03 22:45:02.778722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.778705 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" event={"ID":"85615752-db5a-4dea-bc86-290e793928e0","Type":"ContainerStarted","Data":"a44311d6b9ada1d25d9db52536707c8f69a66c06b3661266a5accaa5f0a35ea0"} Nov 03 22:45:02.779912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.779894 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" event={"ID":"b052a69f-e1a3-408e-9881-94770cb92ed6","Type":"ContainerStarted","Data":"7acc087cfc604a80fab3edfeb7773aca5a6256df384fdb379e0122df17b4a5e4"} Nov 03 22:45:02.781264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.781244 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" event={"ID":"7c71cca5-adfa-4f08-9233-2719cadc9550","Type":"ContainerStarted","Data":"fbddc9beb04d09ce0fa260b7b26cae8f36f60e2c71a4d6e979c4c1edae6a08f6"} Nov 03 22:45:02.799216 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.799163 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" podStartSLOduration=2.799148259 podStartE2EDuration="2.799148259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:02.79393637 +0000 UTC m=+1929.197877344" watchObservedRunningTime="2026-11-03 22:45:02.799148259 +0000 UTC m=+1929.203089234" Nov 03 22:45:02.876368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.876309 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" podStartSLOduration=2.876277589 podStartE2EDuration="2.876277589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:02.870338599 +0000 UTC m=+1929.274279574" watchObservedRunningTime="2026-11-03 22:45:02.876277589 +0000 UTC m=+1929.280218572" Nov 03 22:45:02.908876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:02.908807 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" podStartSLOduration=2.9087855449999998 podStartE2EDuration="2.908785545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:02.905962487 +0000 UTC m=+1929.309903462" watchObservedRunningTime="2026-11-03 22:45:02.908785545 +0000 UTC m=+1929.312726521" Nov 03 22:45:03.203284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.203245 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" podUID="40bf741a-dbda-425a-94b3-27e46a7d1a94" containerName="main" containerID="cri-o://91a5e29516f16afe8da000f51724828c5652344cdbe8fd1e2ddcd554df46a4d4" gracePeriod=45 Nov 03 22:45:03.792975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.792946 4903 generic.go:356] "Generic (PLEG): container finished" podID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerID="8dec5708b48c7ac8500d391ea212ac2a139ca06319f731314c78155e9e42caf9" exitCode=60 Nov 03 22:45:03.793113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.793044 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" event={"ID":"659a4a6e-2d77-4d9f-8284-a29be2d2de56","Type":"ContainerDied","Data":"8dec5708b48c7ac8500d391ea212ac2a139ca06319f731314c78155e9e42caf9"} Nov 03 22:45:03.793517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.793493 4903 scope.go:122] "RemoveContainer" containerID="8dec5708b48c7ac8500d391ea212ac2a139ca06319f731314c78155e9e42caf9" Nov 03 22:45:03.794996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.794974 4903 generic.go:356] "Generic (PLEG): container finished" podID="85615752-db5a-4dea-bc86-290e793928e0" containerID="a44311d6b9ada1d25d9db52536707c8f69a66c06b3661266a5accaa5f0a35ea0" exitCode=60 Nov 03 22:45:03.795090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.795049 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" event={"ID":"85615752-db5a-4dea-bc86-290e793928e0","Type":"ContainerDied","Data":"a44311d6b9ada1d25d9db52536707c8f69a66c06b3661266a5accaa5f0a35ea0"} Nov 03 22:45:03.795381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.795354 4903 scope.go:122] "RemoveContainer" containerID="a44311d6b9ada1d25d9db52536707c8f69a66c06b3661266a5accaa5f0a35ea0" Nov 03 22:45:03.797385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.797363 4903 generic.go:356] "Generic (PLEG): container finished" podID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerID="4a3503f3e5ae885cd20e57d5cb9c00677d55142e38601e3f34b3af05025a2873" exitCode=60 Nov 03 22:45:03.797576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.797450 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" event={"ID":"7c71cca5-adfa-4f08-9233-2719cadc9550","Type":"ContainerDied","Data":"4a3503f3e5ae885cd20e57d5cb9c00677d55142e38601e3f34b3af05025a2873"} Nov 03 22:45:03.797705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.797680 4903 scope.go:122] "RemoveContainer" containerID="4a3503f3e5ae885cd20e57d5cb9c00677d55142e38601e3f34b3af05025a2873" Nov 03 22:45:03.803273 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.803246 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" event={"ID":"1010f884-c3b1-442b-bd6d-848666e3bfb4","Type":"ContainerStarted","Data":"6d1f374375ff1d4e81f6fb1f612c64e65b39811ea7909908730e43ad88a76727"} Nov 03 22:45:03.804872 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.804848 4903 generic.go:356] "Generic (PLEG): container finished" podID="40bf741a-dbda-425a-94b3-27e46a7d1a94" containerID="91a5e29516f16afe8da000f51724828c5652344cdbe8fd1e2ddcd554df46a4d4" exitCode=0 Nov 03 22:45:03.805007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.804915 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" event={"ID":"40bf741a-dbda-425a-94b3-27e46a7d1a94","Type":"ContainerDied","Data":"91a5e29516f16afe8da000f51724828c5652344cdbe8fd1e2ddcd554df46a4d4"} Nov 03 22:45:03.805007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.804960 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" event={"ID":"40bf741a-dbda-425a-94b3-27e46a7d1a94","Type":"ContainerDied","Data":"5aeabcdeac62f2207450d4801b7165142498cb8b9a8d0cde387305c2ee35465f"} Nov 03 22:45:03.805007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.804976 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aeabcdeac62f2207450d4801b7165142498cb8b9a8d0cde387305c2ee35465f" Nov 03 22:45:03.807446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.806946 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" event={"ID":"598378e1-2604-411f-9b3c-e2a196b4dc2e","Type":"ContainerStarted","Data":"1c8d33398277be7b673bfab73347618135bf3a6da609522019b65af7189b0389"} Nov 03 22:45:03.809147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.809114 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" event={"ID":"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164","Type":"ContainerStarted","Data":"2e7736050355eda49c92ddefe69b0ea2d54567167f66c5319586a541728cd20f"} Nov 03 22:45:03.810912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.810879 4903 generic.go:356] "Generic (PLEG): container finished" podID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerID="143d19cf9fcf399143f23ca54a26ad14a0fbe9d08864dae427c91c55fc427b40" exitCode=60 Nov 03 22:45:03.811005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.810921 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" event={"ID":"9445c683-e8c3-41bf-9cbc-5d3093d840c6","Type":"ContainerDied","Data":"143d19cf9fcf399143f23ca54a26ad14a0fbe9d08864dae427c91c55fc427b40"} Nov 03 22:45:03.811207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.811192 4903 scope.go:122] "RemoveContainer" containerID="143d19cf9fcf399143f23ca54a26ad14a0fbe9d08864dae427c91c55fc427b40" Nov 03 22:45:03.812769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.812743 4903 generic.go:356] "Generic (PLEG): container finished" podID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerID="df340e92253953bf9fa6a63be87e4d2e8a130551bb62ee62fe5abbc6688fc4a1" exitCode=60 Nov 03 22:45:03.812885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.812830 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" event={"ID":"3ec653a6-42f1-49b3-a7d9-827945ef8946","Type":"ContainerDied","Data":"df340e92253953bf9fa6a63be87e4d2e8a130551bb62ee62fe5abbc6688fc4a1"} Nov 03 22:45:03.813000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.812985 4903 scope.go:122] "RemoveContainer" containerID="df340e92253953bf9fa6a63be87e4d2e8a130551bb62ee62fe5abbc6688fc4a1" Nov 03 22:45:03.814734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.814711 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" event={"ID":"2f271965-1d01-4f06-8619-ac80aaeb2154","Type":"ContainerStarted","Data":"cd013c130999fdba01b523d4f2e66a2ebf890e4adfee1a481f0c98a591c96940"} Nov 03 22:45:03.816523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.816500 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" event={"ID":"1b686411-6854-4f2a-9400-8632d8e55009","Type":"ContainerStarted","Data":"03ab36c6fb67e4ca403afa4a65a6e475f3f72add2a510b9e6c88361e8dde505a"} Nov 03 22:45:03.822026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.822006 4903 generic.go:356] "Generic (PLEG): container finished" podID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerID="886e2492b53d6e83ebdbefb761ac321bce073c96300bd8e4b73c360c5a3651f1" exitCode=60 Nov 03 22:45:03.822106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.822084 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" event={"ID":"31ad238a-db65-4160-a25b-9e29df9dbc94","Type":"ContainerDied","Data":"886e2492b53d6e83ebdbefb761ac321bce073c96300bd8e4b73c360c5a3651f1"} Nov 03 22:45:03.822381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.822366 4903 scope.go:122] "RemoveContainer" containerID="886e2492b53d6e83ebdbefb761ac321bce073c96300bd8e4b73c360c5a3651f1" Nov 03 22:45:03.823925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.823906 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" event={"ID":"4e3fdb74-d353-4007-a96b-3af3db01cc57","Type":"ContainerStarted","Data":"0a6f07cefdd39abc118596a15df19be7051241ed167549a38604f2f0670118a8"} Nov 03 22:45:03.826519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.826498 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" event={"ID":"c9d6c159-15e1-4fc7-8d50-40758bd126be","Type":"ContainerStarted","Data":"ba8c82e1d37be8dad3f1c178295d11c2db858bee2c193839683d178df5fca984"} Nov 03 22:45:03.828055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.828035 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" event={"ID":"1d9725d5-d50c-496b-834e-a783e0c47b28","Type":"ContainerStarted","Data":"2c6e7a3c2499c03b2ad97b0afa002bb1c947278db9f9cc08cba973dcadc5f193"} Nov 03 22:45:03.829580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.829558 4903 generic.go:356] "Generic (PLEG): container finished" podID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerID="85c28ad5f7b3153940a44fb064481690e86110f8b1f84834bb34bf65699612f9" exitCode=60 Nov 03 22:45:03.829668 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.829636 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" event={"ID":"c63f0f29-03cf-4880-8e34-b44f25a1eb2a","Type":"ContainerDied","Data":"85c28ad5f7b3153940a44fb064481690e86110f8b1f84834bb34bf65699612f9"} Nov 03 22:45:03.829814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.829801 4903 scope.go:122] "RemoveContainer" containerID="85c28ad5f7b3153940a44fb064481690e86110f8b1f84834bb34bf65699612f9" Nov 03 22:45:03.949409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:03.949356 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" podStartSLOduration=3.94933597 podStartE2EDuration="3.94933597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:03.945450738 +0000 UTC m=+1930.349391715" watchObservedRunningTime="2026-11-03 22:45:03.94933597 +0000 UTC m=+1930.353276947" Nov 03 22:45:04.039414 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.039348 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" podStartSLOduration=4.039327639 podStartE2EDuration="4.039327639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.038049802 +0000 UTC m=+1930.441990794" watchObservedRunningTime="2026-11-03 22:45:04.039327639 +0000 UTC m=+1930.443268617" Nov 03 22:45:04.119305 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.119238 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" podStartSLOduration=4.119223401 podStartE2EDuration="4.119223401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.114642396 +0000 UTC m=+1930.518583371" watchObservedRunningTime="2026-11-03 22:45:04.119223401 +0000 UTC m=+1930.523164378" Nov 03 22:45:04.152274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.152211 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" podStartSLOduration=4.152191427 podStartE2EDuration="4.152191427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.149615667 +0000 UTC m=+1930.553556642" watchObservedRunningTime="2026-11-03 22:45:04.152191427 +0000 UTC m=+1930.556132403" Nov 03 22:45:04.259556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.259488 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" podStartSLOduration=4.259468673 podStartE2EDuration="4.259468673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.258372911 +0000 UTC m=+1930.662313887" watchObservedRunningTime="2026-11-03 22:45:04.259468673 +0000 UTC m=+1930.663409651" Nov 03 22:45:04.299035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.298981 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" podStartSLOduration=4.298965445 podStartE2EDuration="4.298965445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.293906309 +0000 UTC m=+1930.697847284" watchObservedRunningTime="2026-11-03 22:45:04.298965445 +0000 UTC m=+1930.702906459" Nov 03 22:45:04.391052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.390981 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" podStartSLOduration=4.390961333 podStartE2EDuration="4.390961333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.384022967 +0000 UTC m=+1930.787963944" watchObservedRunningTime="2026-11-03 22:45:04.390961333 +0000 UTC m=+1930.794902309" Nov 03 22:45:04.458437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.458384 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" podStartSLOduration=4.45837197 podStartE2EDuration="4.45837197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:04.457037001 +0000 UTC m=+1930.860977976" watchObservedRunningTime="2026-11-03 22:45:04.45837197 +0000 UTC m=+1930.862312961" Nov 03 22:45:04.844545 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.844498 4903 generic.go:356] "Generic (PLEG): container finished" podID="1b686411-6854-4f2a-9400-8632d8e55009" containerID="03ab36c6fb67e4ca403afa4a65a6e475f3f72add2a510b9e6c88361e8dde505a" exitCode=60 Nov 03 22:45:04.844732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.844569 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" event={"ID":"1b686411-6854-4f2a-9400-8632d8e55009","Type":"ContainerDied","Data":"03ab36c6fb67e4ca403afa4a65a6e475f3f72add2a510b9e6c88361e8dde505a"} Nov 03 22:45:04.845071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.845053 4903 scope.go:122] "RemoveContainer" containerID="03ab36c6fb67e4ca403afa4a65a6e475f3f72add2a510b9e6c88361e8dde505a" Nov 03 22:45:04.847261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.847236 4903 generic.go:356] "Generic (PLEG): container finished" podID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerID="0a6f07cefdd39abc118596a15df19be7051241ed167549a38604f2f0670118a8" exitCode=60 Nov 03 22:45:04.847454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.847354 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" event={"ID":"4e3fdb74-d353-4007-a96b-3af3db01cc57","Type":"ContainerDied","Data":"0a6f07cefdd39abc118596a15df19be7051241ed167549a38604f2f0670118a8"} Nov 03 22:45:04.847665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.847641 4903 scope.go:122] "RemoveContainer" containerID="0a6f07cefdd39abc118596a15df19be7051241ed167549a38604f2f0670118a8" Nov 03 22:45:04.854778 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.854555 4903 generic.go:356] "Generic (PLEG): container finished" podID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerID="ba8c82e1d37be8dad3f1c178295d11c2db858bee2c193839683d178df5fca984" exitCode=60 Nov 03 22:45:04.854778 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.854645 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" event={"ID":"c9d6c159-15e1-4fc7-8d50-40758bd126be","Type":"ContainerDied","Data":"ba8c82e1d37be8dad3f1c178295d11c2db858bee2c193839683d178df5fca984"} Nov 03 22:45:04.855098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.855078 4903 scope.go:122] "RemoveContainer" containerID="ba8c82e1d37be8dad3f1c178295d11c2db858bee2c193839683d178df5fca984" Nov 03 22:45:04.868045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.868014 4903 generic.go:356] "Generic (PLEG): container finished" podID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerID="2c6e7a3c2499c03b2ad97b0afa002bb1c947278db9f9cc08cba973dcadc5f193" exitCode=60 Nov 03 22:45:04.868182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.868131 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" event={"ID":"1d9725d5-d50c-496b-834e-a783e0c47b28","Type":"ContainerDied","Data":"2c6e7a3c2499c03b2ad97b0afa002bb1c947278db9f9cc08cba973dcadc5f193"} Nov 03 22:45:04.868539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.868514 4903 scope.go:122] "RemoveContainer" containerID="2c6e7a3c2499c03b2ad97b0afa002bb1c947278db9f9cc08cba973dcadc5f193" Nov 03 22:45:04.872530 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.872507 4903 generic.go:356] "Generic (PLEG): container finished" podID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerID="e0f1c96653b0457253ca5101deefb3be0eeffe4a9d178d4f4f6d5dd099e40741" exitCode=60 Nov 03 22:45:04.873055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.872700 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" event={"ID":"b10ff0e7-d241-428a-9e02-ac902f55b8c1","Type":"ContainerDied","Data":"e0f1c96653b0457253ca5101deefb3be0eeffe4a9d178d4f4f6d5dd099e40741"} Nov 03 22:45:04.873055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.872993 4903 scope.go:122] "RemoveContainer" containerID="e0f1c96653b0457253ca5101deefb3be0eeffe4a9d178d4f4f6d5dd099e40741" Nov 03 22:45:04.874965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.874940 4903 generic.go:356] "Generic (PLEG): container finished" podID="663b664e-98af-4583-aed0-da9572e83f52" containerID="7e104337abdb86b4262ffb022b5415c1613578e2899c52e1bb888d9ba09fea5b" exitCode=60 Nov 03 22:45:04.875084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.875068 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" event={"ID":"663b664e-98af-4583-aed0-da9572e83f52","Type":"ContainerDied","Data":"7e104337abdb86b4262ffb022b5415c1613578e2899c52e1bb888d9ba09fea5b"} Nov 03 22:45:04.875327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.875271 4903 scope.go:122] "RemoveContainer" containerID="7e104337abdb86b4262ffb022b5415c1613578e2899c52e1bb888d9ba09fea5b" Nov 03 22:45:04.879031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.879004 4903 generic.go:356] "Generic (PLEG): container finished" podID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerID="136fbabc17d6a9ba3fa043de1dcffd7aa8e4d772f7564c96de6b7d236d3fe731" exitCode=60 Nov 03 22:45:04.879143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.879081 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" event={"ID":"d177e30a-9ef2-4d4d-a2be-3872a6bc9411","Type":"ContainerDied","Data":"136fbabc17d6a9ba3fa043de1dcffd7aa8e4d772f7564c96de6b7d236d3fe731"} Nov 03 22:45:04.879799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.879484 4903 scope.go:122] "RemoveContainer" containerID="136fbabc17d6a9ba3fa043de1dcffd7aa8e4d772f7564c96de6b7d236d3fe731" Nov 03 22:45:04.894501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.889105 4903 generic.go:356] "Generic (PLEG): container finished" podID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerID="40abb25d43764f97ea2930c68598a64c312122d8503f21542e0f0dc90a114382" exitCode=60 Nov 03 22:45:04.894501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.889237 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" event={"ID":"b052a69f-e1a3-408e-9881-94770cb92ed6","Type":"ContainerDied","Data":"40abb25d43764f97ea2930c68598a64c312122d8503f21542e0f0dc90a114382"} Nov 03 22:45:04.894501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.889718 4903 scope.go:122] "RemoveContainer" containerID="40abb25d43764f97ea2930c68598a64c312122d8503f21542e0f0dc90a114382" Nov 03 22:45:04.894501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.892406 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:45:04.896436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.895985 4903 generic.go:356] "Generic (PLEG): container finished" podID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerID="6d1f374375ff1d4e81f6fb1f612c64e65b39811ea7909908730e43ad88a76727" exitCode=60 Nov 03 22:45:04.896436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.896078 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" event={"ID":"1010f884-c3b1-442b-bd6d-848666e3bfb4","Type":"ContainerDied","Data":"6d1f374375ff1d4e81f6fb1f612c64e65b39811ea7909908730e43ad88a76727"} Nov 03 22:45:04.896436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.896414 4903 scope.go:122] "RemoveContainer" containerID="6d1f374375ff1d4e81f6fb1f612c64e65b39811ea7909908730e43ad88a76727" Nov 03 22:45:04.899074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.899052 4903 generic.go:356] "Generic (PLEG): container finished" podID="fb962f11-d246-41e8-a596-6383a2d91a58" containerID="c4f01b9a342d4453b4a2c6fcfece5e609661f6f6fe722590532e9a6ff7040778" exitCode=60 Nov 03 22:45:04.899235 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.899218 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" event={"ID":"fb962f11-d246-41e8-a596-6383a2d91a58","Type":"ContainerDied","Data":"c4f01b9a342d4453b4a2c6fcfece5e609661f6f6fe722590532e9a6ff7040778"} Nov 03 22:45:04.899593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.899575 4903 scope.go:122] "RemoveContainer" containerID="c4f01b9a342d4453b4a2c6fcfece5e609661f6f6fe722590532e9a6ff7040778" Nov 03 22:45:04.901518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.901496 4903 generic.go:356] "Generic (PLEG): container finished" podID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerID="1c8d33398277be7b673bfab73347618135bf3a6da609522019b65af7189b0389" exitCode=60 Nov 03 22:45:04.901630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.901589 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" event={"ID":"598378e1-2604-411f-9b3c-e2a196b4dc2e","Type":"ContainerDied","Data":"1c8d33398277be7b673bfab73347618135bf3a6da609522019b65af7189b0389"} Nov 03 22:45:04.901969 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.901946 4903 scope.go:122] "RemoveContainer" containerID="1c8d33398277be7b673bfab73347618135bf3a6da609522019b65af7189b0389" Nov 03 22:45:04.909105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.909079 4903 generic.go:356] "Generic (PLEG): container finished" podID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerID="2e7736050355eda49c92ddefe69b0ea2d54567167f66c5319586a541728cd20f" exitCode=60 Nov 03 22:45:04.909202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.909183 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" event={"ID":"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164","Type":"ContainerDied","Data":"2e7736050355eda49c92ddefe69b0ea2d54567167f66c5319586a541728cd20f"} Nov 03 22:45:04.909608 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.909586 4903 scope.go:122] "RemoveContainer" containerID="2e7736050355eda49c92ddefe69b0ea2d54567167f66c5319586a541728cd20f" Nov 03 22:45:04.913855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.913829 4903 generic.go:356] "Generic (PLEG): container finished" podID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerID="cd013c130999fdba01b523d4f2e66a2ebf890e4adfee1a481f0c98a591c96940" exitCode=60 Nov 03 22:45:04.913971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.913882 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" event={"ID":"2f271965-1d01-4f06-8619-ac80aaeb2154","Type":"ContainerDied","Data":"cd013c130999fdba01b523d4f2e66a2ebf890e4adfee1a481f0c98a591c96940"} Nov 03 22:45:04.914230 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.914208 4903 scope.go:122] "RemoveContainer" containerID="cd013c130999fdba01b523d4f2e66a2ebf890e4adfee1a481f0c98a591c96940" Nov 03 22:45:04.989666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.987801 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-model-cache\") pod \"40bf741a-dbda-425a-94b3-27e46a7d1a94\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " Nov 03 22:45:04.989666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.987850 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40bf741a-dbda-425a-94b3-27e46a7d1a94-tls-certs\" (UniqueName: \"kubernetes.io/secret/40bf741a-dbda-425a-94b3-27e46a7d1a94-tls-certs\") pod \"40bf741a-dbda-425a-94b3-27e46a7d1a94\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " Nov 03 22:45:04.989666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.988203 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-home\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-home\") pod \"40bf741a-dbda-425a-94b3-27e46a7d1a94\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " Nov 03 22:45:04.989666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.988319 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-dshm\") pod \"40bf741a-dbda-425a-94b3-27e46a7d1a94\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " Nov 03 22:45:04.989666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.988413 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-tmp-dir\") pod \"40bf741a-dbda-425a-94b3-27e46a7d1a94\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " Nov 03 22:45:04.989666 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.988496 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bf741a-dbda-425a-94b3-27e46a7d1a94-kube-api-access-z6crq\" (UniqueName: \"kubernetes.io/projected/40bf741a-dbda-425a-94b3-27e46a7d1a94-kube-api-access-z6crq\") pod \"40bf741a-dbda-425a-94b3-27e46a7d1a94\" (UID: \"40bf741a-dbda-425a-94b3-27e46a7d1a94\") " Nov 03 22:45:04.990140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.990062 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-model-cache" pod "40bf741a-dbda-425a-94b3-27e46a7d1a94" (UID: "40bf741a-dbda-425a-94b3-27e46a7d1a94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:04.993518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.991526 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-home" pod "40bf741a-dbda-425a-94b3-27e46a7d1a94" (UID: "40bf741a-dbda-425a-94b3-27e46a7d1a94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:04.993518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.992259 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:04.993518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.992279 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:04.993518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.992487 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-tmp-dir" pod "40bf741a-dbda-425a-94b3-27e46a7d1a94" (UID: "40bf741a-dbda-425a-94b3-27e46a7d1a94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:04.994107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.994079 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40bf741a-dbda-425a-94b3-27e46a7d1a94-tls-certs" pod "40bf741a-dbda-425a-94b3-27e46a7d1a94" (UID: "40bf741a-dbda-425a-94b3-27e46a7d1a94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:04.994214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.994184 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-dshm" pod "40bf741a-dbda-425a-94b3-27e46a7d1a94" (UID: "40bf741a-dbda-425a-94b3-27e46a7d1a94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:04.996399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:04.996270 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bf741a-dbda-425a-94b3-27e46a7d1a94-kube-api-access-z6crq" pod "40bf741a-dbda-425a-94b3-27e46a7d1a94" (UID: "40bf741a-dbda-425a-94b3-27e46a7d1a94"). InnerVolumeSpecName "kube-api-access-z6crq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:05.094133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.094102 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:05.094133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.094129 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bf741a-dbda-425a-94b3-27e46a7d1a94-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:05.094133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.094138 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6crq\" (UniqueName: \"kubernetes.io/projected/40bf741a-dbda-425a-94b3-27e46a7d1a94-kube-api-access-z6crq\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:05.094133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.094147 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bf741a-dbda-425a-94b3-27e46a7d1a94-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:05.249483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.249442 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" podUID="40bf741a-dbda-425a-94b3-27e46a7d1a94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 22:45:05.952122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.951785 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" event={"ID":"659a4a6e-2d77-4d9f-8284-a29be2d2de56","Type":"ContainerStarted","Data":"3040426ac54dd251b5e5ca081b54b45d23c3a2d6a107f4ae6e46f77621491ec3"} Nov 03 22:45:05.956581 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.956542 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" event={"ID":"d177e30a-9ef2-4d4d-a2be-3872a6bc9411","Type":"ContainerStarted","Data":"b9853b451bc09b7a0b9d47a6bd5a432e8114df42f87d4a60f6814dc706bcf7c1"} Nov 03 22:45:05.972392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.972038 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" event={"ID":"85615752-db5a-4dea-bc86-290e793928e0","Type":"ContainerStarted","Data":"b88f4818986ec455b0d0a9cd309803b0f0ae97aef0607b5351a0fe60f77405bf"} Nov 03 22:45:05.975675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.975643 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" event={"ID":"7c71cca5-adfa-4f08-9233-2719cadc9550","Type":"ContainerStarted","Data":"86434e39ef5458fe446d09f9df003b6091404d556697220e2df4c6b8992fe064"} Nov 03 22:45:05.978891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.978861 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" event={"ID":"1010f884-c3b1-442b-bd6d-848666e3bfb4","Type":"ContainerStarted","Data":"b48831d65fc923bbcbb377a8026c1d6eac310556cbfcbfac414e94b495314c39"} Nov 03 22:45:05.982815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.982517 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" event={"ID":"598378e1-2604-411f-9b3c-e2a196b4dc2e","Type":"ContainerStarted","Data":"a671baa0aa3d368f211fcbff25b2bd1345b1e47d8732a99fec4b8af6888966b5"} Nov 03 22:45:05.984991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.984959 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" event={"ID":"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164","Type":"ContainerStarted","Data":"028c09f8ff7f38e8b0a631ea28f8ab00a8f8e29485b0409f8b8e5b46edce0d28"} Nov 03 22:45:05.997640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:05.997607 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" event={"ID":"9445c683-e8c3-41bf-9cbc-5d3093d840c6","Type":"ContainerStarted","Data":"e4c35972457370c4818106312d3b47db41ad42facfb2584e8a365f6964a23288"} Nov 03 22:45:06.007340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.007261 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" event={"ID":"3ec653a6-42f1-49b3-a7d9-827945ef8946","Type":"ContainerStarted","Data":"9ef92eaa7acdd3a7b4ec388df510294a2db2296620866929e5d7f46296e22ee3"} Nov 03 22:45:06.009775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.009742 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" event={"ID":"1b686411-6854-4f2a-9400-8632d8e55009","Type":"ContainerStarted","Data":"d8726ab9b34a419f4463dbafeccff9865592c62119cab7df872452fab0707ed5"} Nov 03 22:45:06.012140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.012114 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" event={"ID":"31ad238a-db65-4160-a25b-9e29df9dbc94","Type":"ContainerStarted","Data":"43358d3bc9debb1679337cf29be821b4a2bc01f496dd7d01317366a582ff7380"} Nov 03 22:45:06.014525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.014467 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" event={"ID":"4e3fdb74-d353-4007-a96b-3af3db01cc57","Type":"ContainerStarted","Data":"61f9aab06d7e39cc3ebabeab31aa3e47c05b6617a55c9e1fa04a450280e1a2fc"} Nov 03 22:45:06.017441 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.017417 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" event={"ID":"c9d6c159-15e1-4fc7-8d50-40758bd126be","Type":"ContainerStarted","Data":"5ac015103701def7a9eac439e282a47193b9adf15b9e6ed97edd79af5cffba83"} Nov 03 22:45:06.021008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.020973 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" event={"ID":"1d9725d5-d50c-496b-834e-a783e0c47b28","Type":"ContainerStarted","Data":"70b5fd2693aa61533c4e7e588e490cdef597a668f19a6626bdb9aaef6fc5b078"} Nov 03 22:45:06.023125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.023097 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" event={"ID":"c63f0f29-03cf-4880-8e34-b44f25a1eb2a","Type":"ContainerStarted","Data":"78a5d306823b20bc8a2ece7fda9266f8ad1651d2a90449123993ba583b1912d2"} Nov 03 22:45:06.025136 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.025110 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" event={"ID":"b10ff0e7-d241-428a-9e02-ac902f55b8c1","Type":"ContainerStarted","Data":"4b98ad0cb672aba4321624a9249554ddc3d995c2f6eb689bc7c2e98f120d1aba"} Nov 03 22:45:06.027145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.027117 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" event={"ID":"663b664e-98af-4583-aed0-da9572e83f52","Type":"ContainerStarted","Data":"5427e22b12520acb7a80a3dc26c037e893eef49ea254b17ffc84618fa971c2e1"} Nov 03 22:45:06.027233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.027164 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2" Nov 03 22:45:06.039878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.039816 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" podStartSLOduration=6.039795695 podStartE2EDuration="6.039795695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:05.997005352 +0000 UTC m=+1932.400946339" watchObservedRunningTime="2026-11-03 22:45:06.039795695 +0000 UTC m=+1932.443736673" Nov 03 22:45:06.079208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.079140 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" podStartSLOduration=6.079119514 podStartE2EDuration="6.079119514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.075833166 +0000 UTC m=+1932.479774140" watchObservedRunningTime="2026-11-03 22:45:06.079119514 +0000 UTC m=+1932.483060491" Nov 03 22:45:06.168870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.168816 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" podStartSLOduration=6.16880191 podStartE2EDuration="6.16880191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.165917415 +0000 UTC m=+1932.569858415" watchObservedRunningTime="2026-11-03 22:45:06.16880191 +0000 UTC m=+1932.572742884" Nov 03 22:45:06.314035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.314005 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2"] Nov 03 22:45:06.319370 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.319344 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4b5e2b/test-model-4b5e2b-kserve-774dff4587-kdzz2"] Nov 03 22:45:06.358640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.357671 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" podStartSLOduration=6.357651133 podStartE2EDuration="6.357651133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.35526791 +0000 UTC m=+1932.759208886" watchObservedRunningTime="2026-11-03 22:45:06.357651133 +0000 UTC m=+1932.761592109" Nov 03 22:45:06.432603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.432549 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" podStartSLOduration=6.4325325509999995 podStartE2EDuration="6.432532551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.427484065 +0000 UTC m=+1932.831425040" watchObservedRunningTime="2026-11-03 22:45:06.432532551 +0000 UTC m=+1932.836473550" Nov 03 22:45:06.510069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.510015 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" podStartSLOduration=6.509998317 podStartE2EDuration="6.509998317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.50494764 +0000 UTC m=+1932.908888617" watchObservedRunningTime="2026-11-03 22:45:06.509998317 +0000 UTC m=+1932.913939313" Nov 03 22:45:06.583505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:06.583441 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" podStartSLOduration=6.583423096 podStartE2EDuration="6.583423096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:06.58255087 +0000 UTC m=+1932.986491846" watchObservedRunningTime="2026-11-03 22:45:06.583423096 +0000 UTC m=+1932.987364072" Nov 03 22:45:07.039881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.039839 4903 generic.go:356] "Generic (PLEG): container finished" podID="fb962f11-d246-41e8-a596-6383a2d91a58" containerID="53d7e5166594cdc40ff7ccee53cb6e6ac458a4645884f5079bc8e33dfefac01d" exitCode=60 Nov 03 22:45:07.040399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.039919 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" event={"ID":"fb962f11-d246-41e8-a596-6383a2d91a58","Type":"ContainerDied","Data":"53d7e5166594cdc40ff7ccee53cb6e6ac458a4645884f5079bc8e33dfefac01d"} Nov 03 22:45:07.040399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.039982 4903 scope.go:122] "RemoveContainer" containerID="c4f01b9a342d4453b4a2c6fcfece5e609661f6f6fe722590532e9a6ff7040778" Nov 03 22:45:07.040399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.040381 4903 scope.go:122] "RemoveContainer" containerID="53d7e5166594cdc40ff7ccee53cb6e6ac458a4645884f5079bc8e33dfefac01d" Nov 03 22:45:07.040822 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.040799 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p_odh-ai-gateway-infra(fb962f11-d246-41e8-a596-6383a2d91a58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" Nov 03 22:45:07.043239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.043214 4903 generic.go:356] "Generic (PLEG): container finished" podID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerID="a671baa0aa3d368f211fcbff25b2bd1345b1e47d8732a99fec4b8af6888966b5" exitCode=60 Nov 03 22:45:07.043518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.043388 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" event={"ID":"598378e1-2604-411f-9b3c-e2a196b4dc2e","Type":"ContainerDied","Data":"a671baa0aa3d368f211fcbff25b2bd1345b1e47d8732a99fec4b8af6888966b5"} Nov 03 22:45:07.043684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.043663 4903 scope.go:122] "RemoveContainer" containerID="a671baa0aa3d368f211fcbff25b2bd1345b1e47d8732a99fec4b8af6888966b5" Nov 03 22:45:07.044088 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.044066 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d_odh-ai-gateway-infra(598378e1-2604-411f-9b3c-e2a196b4dc2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" Nov 03 22:45:07.045618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.045592 4903 generic.go:356] "Generic (PLEG): container finished" podID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerID="028c09f8ff7f38e8b0a631ea28f8ab00a8f8e29485b0409f8b8e5b46edce0d28" exitCode=60 Nov 03 22:45:07.045727 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.045637 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" event={"ID":"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164","Type":"ContainerDied","Data":"028c09f8ff7f38e8b0a631ea28f8ab00a8f8e29485b0409f8b8e5b46edce0d28"} Nov 03 22:45:07.045980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.045964 4903 scope.go:122] "RemoveContainer" containerID="028c09f8ff7f38e8b0a631ea28f8ab00a8f8e29485b0409f8b8e5b46edce0d28" Nov 03 22:45:07.046377 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.046353 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q_odh-ai-gateway-infra(3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" Nov 03 22:45:07.047924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.047901 4903 generic.go:356] "Generic (PLEG): container finished" podID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerID="e4c35972457370c4818106312d3b47db41ad42facfb2584e8a365f6964a23288" exitCode=60 Nov 03 22:45:07.048067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.047986 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" event={"ID":"9445c683-e8c3-41bf-9cbc-5d3093d840c6","Type":"ContainerDied","Data":"e4c35972457370c4818106312d3b47db41ad42facfb2584e8a365f6964a23288"} Nov 03 22:45:07.048593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.048572 4903 scope.go:122] "RemoveContainer" containerID="e4c35972457370c4818106312d3b47db41ad42facfb2584e8a365f6964a23288" Nov 03 22:45:07.049066 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.049045 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5_odh-ai-gateway-infra(9445c683-e8c3-41bf-9cbc-5d3093d840c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" Nov 03 22:45:07.050341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.050316 4903 generic.go:356] "Generic (PLEG): container finished" podID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerID="9ef92eaa7acdd3a7b4ec388df510294a2db2296620866929e5d7f46296e22ee3" exitCode=60 Nov 03 22:45:07.050341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.050346 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" event={"ID":"3ec653a6-42f1-49b3-a7d9-827945ef8946","Type":"ContainerDied","Data":"9ef92eaa7acdd3a7b4ec388df510294a2db2296620866929e5d7f46296e22ee3"} Nov 03 22:45:07.050645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.050630 4903 scope.go:122] "RemoveContainer" containerID="9ef92eaa7acdd3a7b4ec388df510294a2db2296620866929e5d7f46296e22ee3" Nov 03 22:45:07.050980 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.050943 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps_odh-ai-gateway-infra(3ec653a6-42f1-49b3-a7d9-827945ef8946)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" Nov 03 22:45:07.052888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.052866 4903 generic.go:356] "Generic (PLEG): container finished" podID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerID="c05195136efc5519a63d72bd375cf87a40804e9b45707df7c5470ddbdb776541" exitCode=60 Nov 03 22:45:07.053045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.052919 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" event={"ID":"2f271965-1d01-4f06-8619-ac80aaeb2154","Type":"ContainerDied","Data":"c05195136efc5519a63d72bd375cf87a40804e9b45707df7c5470ddbdb776541"} Nov 03 22:45:07.053205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.053188 4903 scope.go:122] "RemoveContainer" containerID="c05195136efc5519a63d72bd375cf87a40804e9b45707df7c5470ddbdb776541" Nov 03 22:45:07.053553 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.053528 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn_odh-ai-gateway-infra(2f271965-1d01-4f06-8619-ac80aaeb2154)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" Nov 03 22:45:07.055555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.055535 4903 generic.go:356] "Generic (PLEG): container finished" podID="1b686411-6854-4f2a-9400-8632d8e55009" containerID="d8726ab9b34a419f4463dbafeccff9865592c62119cab7df872452fab0707ed5" exitCode=60 Nov 03 22:45:07.055855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.055751 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" event={"ID":"1b686411-6854-4f2a-9400-8632d8e55009","Type":"ContainerDied","Data":"d8726ab9b34a419f4463dbafeccff9865592c62119cab7df872452fab0707ed5"} Nov 03 22:45:07.055927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.055901 4903 scope.go:122] "RemoveContainer" containerID="d8726ab9b34a419f4463dbafeccff9865592c62119cab7df872452fab0707ed5" Nov 03 22:45:07.056365 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.056264 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq_odh-ai-gateway-infra(1b686411-6854-4f2a-9400-8632d8e55009)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" podUID="1b686411-6854-4f2a-9400-8632d8e55009" Nov 03 22:45:07.058039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.058015 4903 generic.go:356] "Generic (PLEG): container finished" podID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerID="43358d3bc9debb1679337cf29be821b4a2bc01f496dd7d01317366a582ff7380" exitCode=60 Nov 03 22:45:07.058142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.058093 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" event={"ID":"31ad238a-db65-4160-a25b-9e29df9dbc94","Type":"ContainerDied","Data":"43358d3bc9debb1679337cf29be821b4a2bc01f496dd7d01317366a582ff7380"} Nov 03 22:45:07.058344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.058326 4903 scope.go:122] "RemoveContainer" containerID="43358d3bc9debb1679337cf29be821b4a2bc01f496dd7d01317366a582ff7380" Nov 03 22:45:07.058641 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.058617 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r_odh-ai-gateway-infra(31ad238a-db65-4160-a25b-9e29df9dbc94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" Nov 03 22:45:07.060492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.060456 4903 generic.go:356] "Generic (PLEG): container finished" podID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerID="61f9aab06d7e39cc3ebabeab31aa3e47c05b6617a55c9e1fa04a450280e1a2fc" exitCode=60 Nov 03 22:45:07.060601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.060513 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" event={"ID":"4e3fdb74-d353-4007-a96b-3af3db01cc57","Type":"ContainerDied","Data":"61f9aab06d7e39cc3ebabeab31aa3e47c05b6617a55c9e1fa04a450280e1a2fc"} Nov 03 22:45:07.060812 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.060794 4903 scope.go:122] "RemoveContainer" containerID="61f9aab06d7e39cc3ebabeab31aa3e47c05b6617a55c9e1fa04a450280e1a2fc" Nov 03 22:45:07.061098 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.061065 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62_odh-ai-gateway-infra(4e3fdb74-d353-4007-a96b-3af3db01cc57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" Nov 03 22:45:07.063985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.063965 4903 generic.go:356] "Generic (PLEG): container finished" podID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerID="5ac015103701def7a9eac439e282a47193b9adf15b9e6ed97edd79af5cffba83" exitCode=60 Nov 03 22:45:07.064083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.064046 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" event={"ID":"c9d6c159-15e1-4fc7-8d50-40758bd126be","Type":"ContainerDied","Data":"5ac015103701def7a9eac439e282a47193b9adf15b9e6ed97edd79af5cffba83"} Nov 03 22:45:07.064222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.064209 4903 scope.go:122] "RemoveContainer" containerID="5ac015103701def7a9eac439e282a47193b9adf15b9e6ed97edd79af5cffba83" Nov 03 22:45:07.064533 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.064483 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg_odh-ai-gateway-infra(c9d6c159-15e1-4fc7-8d50-40758bd126be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" Nov 03 22:45:07.066067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.066043 4903 generic.go:356] "Generic (PLEG): container finished" podID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerID="70b5fd2693aa61533c4e7e588e490cdef597a668f19a6626bdb9aaef6fc5b078" exitCode=60 Nov 03 22:45:07.066163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.066118 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" event={"ID":"1d9725d5-d50c-496b-834e-a783e0c47b28","Type":"ContainerDied","Data":"70b5fd2693aa61533c4e7e588e490cdef597a668f19a6626bdb9aaef6fc5b078"} Nov 03 22:45:07.066343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.066326 4903 scope.go:122] "RemoveContainer" containerID="70b5fd2693aa61533c4e7e588e490cdef597a668f19a6626bdb9aaef6fc5b078" Nov 03 22:45:07.066579 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.066562 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc_odh-ai-gateway-infra(1d9725d5-d50c-496b-834e-a783e0c47b28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" Nov 03 22:45:07.068111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.068093 4903 generic.go:356] "Generic (PLEG): container finished" podID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerID="78a5d306823b20bc8a2ece7fda9266f8ad1651d2a90449123993ba583b1912d2" exitCode=60 Nov 03 22:45:07.068215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.068155 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" event={"ID":"c63f0f29-03cf-4880-8e34-b44f25a1eb2a","Type":"ContainerDied","Data":"78a5d306823b20bc8a2ece7fda9266f8ad1651d2a90449123993ba583b1912d2"} Nov 03 22:45:07.068391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.068374 4903 scope.go:122] "RemoveContainer" containerID="78a5d306823b20bc8a2ece7fda9266f8ad1651d2a90449123993ba583b1912d2" Nov 03 22:45:07.068621 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.068603 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p_odh-ai-gateway-infra(c63f0f29-03cf-4880-8e34-b44f25a1eb2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" Nov 03 22:45:07.070033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.070013 4903 generic.go:356] "Generic (PLEG): container finished" podID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerID="4b98ad0cb672aba4321624a9249554ddc3d995c2f6eb689bc7c2e98f120d1aba" exitCode=60 Nov 03 22:45:07.070121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.070090 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" event={"ID":"b10ff0e7-d241-428a-9e02-ac902f55b8c1","Type":"ContainerDied","Data":"4b98ad0cb672aba4321624a9249554ddc3d995c2f6eb689bc7c2e98f120d1aba"} Nov 03 22:45:07.070270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.070254 4903 scope.go:122] "RemoveContainer" containerID="4b98ad0cb672aba4321624a9249554ddc3d995c2f6eb689bc7c2e98f120d1aba" Nov 03 22:45:07.070521 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.070498 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr_odh-ai-gateway-infra(b10ff0e7-d241-428a-9e02-ac902f55b8c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" Nov 03 22:45:07.072106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.072087 4903 generic.go:356] "Generic (PLEG): container finished" podID="663b664e-98af-4583-aed0-da9572e83f52" containerID="5427e22b12520acb7a80a3dc26c037e893eef49ea254b17ffc84618fa971c2e1" exitCode=60 Nov 03 22:45:07.072193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.072151 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" event={"ID":"663b664e-98af-4583-aed0-da9572e83f52","Type":"ContainerDied","Data":"5427e22b12520acb7a80a3dc26c037e893eef49ea254b17ffc84618fa971c2e1"} Nov 03 22:45:07.072359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.072343 4903 scope.go:122] "RemoveContainer" containerID="5427e22b12520acb7a80a3dc26c037e893eef49ea254b17ffc84618fa971c2e1" Nov 03 22:45:07.072590 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.072560 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj_odh-ai-gateway-infra(663b664e-98af-4583-aed0-da9572e83f52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" podUID="663b664e-98af-4583-aed0-da9572e83f52" Nov 03 22:45:07.074243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.074224 4903 generic.go:356] "Generic (PLEG): container finished" podID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerID="3040426ac54dd251b5e5ca081b54b45d23c3a2d6a107f4ae6e46f77621491ec3" exitCode=60 Nov 03 22:45:07.074359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.074326 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" event={"ID":"659a4a6e-2d77-4d9f-8284-a29be2d2de56","Type":"ContainerDied","Data":"3040426ac54dd251b5e5ca081b54b45d23c3a2d6a107f4ae6e46f77621491ec3"} Nov 03 22:45:07.074509 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.074491 4903 scope.go:122] "RemoveContainer" containerID="3040426ac54dd251b5e5ca081b54b45d23c3a2d6a107f4ae6e46f77621491ec3" Nov 03 22:45:07.074733 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.074716 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s_odh-ai-gateway-infra(659a4a6e-2d77-4d9f-8284-a29be2d2de56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" Nov 03 22:45:07.076387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.076366 4903 generic.go:356] "Generic (PLEG): container finished" podID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerID="b9853b451bc09b7a0b9d47a6bd5a432e8114df42f87d4a60f6814dc706bcf7c1" exitCode=60 Nov 03 22:45:07.076495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.076448 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" event={"ID":"d177e30a-9ef2-4d4d-a2be-3872a6bc9411","Type":"ContainerDied","Data":"b9853b451bc09b7a0b9d47a6bd5a432e8114df42f87d4a60f6814dc706bcf7c1"} Nov 03 22:45:07.076722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.076705 4903 scope.go:122] "RemoveContainer" containerID="b9853b451bc09b7a0b9d47a6bd5a432e8114df42f87d4a60f6814dc706bcf7c1" Nov 03 22:45:07.076984 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.076958 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw_odh-ai-gateway-infra(d177e30a-9ef2-4d4d-a2be-3872a6bc9411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" Nov 03 22:45:07.078768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.078746 4903 generic.go:356] "Generic (PLEG): container finished" podID="85615752-db5a-4dea-bc86-290e793928e0" containerID="b88f4818986ec455b0d0a9cd309803b0f0ae97aef0607b5351a0fe60f77405bf" exitCode=60 Nov 03 22:45:07.078882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.078806 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" event={"ID":"85615752-db5a-4dea-bc86-290e793928e0","Type":"ContainerDied","Data":"b88f4818986ec455b0d0a9cd309803b0f0ae97aef0607b5351a0fe60f77405bf"} Nov 03 22:45:07.079091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.079077 4903 scope.go:122] "RemoveContainer" containerID="b88f4818986ec455b0d0a9cd309803b0f0ae97aef0607b5351a0fe60f77405bf" Nov 03 22:45:07.079345 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.079289 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp_odh-ai-gateway-infra(85615752-db5a-4dea-bc86-290e793928e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" podUID="85615752-db5a-4dea-bc86-290e793928e0" Nov 03 22:45:07.081007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.080989 4903 generic.go:356] "Generic (PLEG): container finished" podID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerID="b5cd8682f0f58c75bf1d25530ab6f266dee97cb2283859bd0eba1eff81230dff" exitCode=60 Nov 03 22:45:07.081073 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.081039 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" event={"ID":"b052a69f-e1a3-408e-9881-94770cb92ed6","Type":"ContainerDied","Data":"b5cd8682f0f58c75bf1d25530ab6f266dee97cb2283859bd0eba1eff81230dff"} Nov 03 22:45:07.081351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.081336 4903 scope.go:122] "RemoveContainer" containerID="b5cd8682f0f58c75bf1d25530ab6f266dee97cb2283859bd0eba1eff81230dff" Nov 03 22:45:07.081547 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.081529 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4_odh-ai-gateway-infra(b052a69f-e1a3-408e-9881-94770cb92ed6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" Nov 03 22:45:07.083028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.083012 4903 generic.go:356] "Generic (PLEG): container finished" podID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerID="86434e39ef5458fe446d09f9df003b6091404d556697220e2df4c6b8992fe064" exitCode=60 Nov 03 22:45:07.083116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.083105 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" event={"ID":"7c71cca5-adfa-4f08-9233-2719cadc9550","Type":"ContainerDied","Data":"86434e39ef5458fe446d09f9df003b6091404d556697220e2df4c6b8992fe064"} Nov 03 22:45:07.084228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.084209 4903 scope.go:122] "RemoveContainer" containerID="86434e39ef5458fe446d09f9df003b6091404d556697220e2df4c6b8992fe064" Nov 03 22:45:07.084522 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.084501 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh_odh-ai-gateway-infra(7c71cca5-adfa-4f08-9233-2719cadc9550)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" Nov 03 22:45:07.086638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.086610 4903 generic.go:356] "Generic (PLEG): container finished" podID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerID="b48831d65fc923bbcbb377a8026c1d6eac310556cbfcbfac414e94b495314c39" exitCode=60 Nov 03 22:45:07.086741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.086665 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" event={"ID":"1010f884-c3b1-442b-bd6d-848666e3bfb4","Type":"ContainerDied","Data":"b48831d65fc923bbcbb377a8026c1d6eac310556cbfcbfac414e94b495314c39"} Nov 03 22:45:07.087066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.087042 4903 scope.go:122] "RemoveContainer" containerID="b48831d65fc923bbcbb377a8026c1d6eac310556cbfcbfac414e94b495314c39" Nov 03 22:45:07.087882 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:07.087855 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv_odh-ai-gateway-infra(1010f884-c3b1-442b-bd6d-848666e3bfb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" Nov 03 22:45:07.526425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.526401 4903 scope.go:122] "RemoveContainer" containerID="1c8d33398277be7b673bfab73347618135bf3a6da609522019b65af7189b0389" Nov 03 22:45:07.606127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.606103 4903 scope.go:122] "RemoveContainer" containerID="2e7736050355eda49c92ddefe69b0ea2d54567167f66c5319586a541728cd20f" Nov 03 22:45:07.623772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.623750 4903 scope.go:122] "RemoveContainer" containerID="143d19cf9fcf399143f23ca54a26ad14a0fbe9d08864dae427c91c55fc427b40" Nov 03 22:45:07.639557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.639537 4903 scope.go:122] "RemoveContainer" containerID="df340e92253953bf9fa6a63be87e4d2e8a130551bb62ee62fe5abbc6688fc4a1" Nov 03 22:45:07.654239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.654219 4903 scope.go:122] "RemoveContainer" containerID="cd013c130999fdba01b523d4f2e66a2ebf890e4adfee1a481f0c98a591c96940" Nov 03 22:45:07.668744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.668723 4903 scope.go:122] "RemoveContainer" containerID="03ab36c6fb67e4ca403afa4a65a6e475f3f72add2a510b9e6c88361e8dde505a" Nov 03 22:45:07.682439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.682420 4903 scope.go:122] "RemoveContainer" containerID="886e2492b53d6e83ebdbefb761ac321bce073c96300bd8e4b73c360c5a3651f1" Nov 03 22:45:07.696038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.696017 4903 scope.go:122] "RemoveContainer" containerID="0a6f07cefdd39abc118596a15df19be7051241ed167549a38604f2f0670118a8" Nov 03 22:45:07.711818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.711798 4903 scope.go:122] "RemoveContainer" containerID="ba8c82e1d37be8dad3f1c178295d11c2db858bee2c193839683d178df5fca984" Nov 03 22:45:07.725383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.725361 4903 scope.go:122] "RemoveContainer" containerID="2c6e7a3c2499c03b2ad97b0afa002bb1c947278db9f9cc08cba973dcadc5f193" Nov 03 22:45:07.741637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.741606 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bf741a-dbda-425a-94b3-27e46a7d1a94" path="/var/lib/kubelet/pods/40bf741a-dbda-425a-94b3-27e46a7d1a94/volumes" Nov 03 22:45:07.741724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.741640 4903 scope.go:122] "RemoveContainer" containerID="85c28ad5f7b3153940a44fb064481690e86110f8b1f84834bb34bf65699612f9" Nov 03 22:45:07.757484 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.757459 4903 scope.go:122] "RemoveContainer" containerID="e0f1c96653b0457253ca5101deefb3be0eeffe4a9d178d4f4f6d5dd099e40741" Nov 03 22:45:07.771149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.771126 4903 scope.go:122] "RemoveContainer" containerID="7e104337abdb86b4262ffb022b5415c1613578e2899c52e1bb888d9ba09fea5b" Nov 03 22:45:07.785669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.785640 4903 scope.go:122] "RemoveContainer" containerID="8dec5708b48c7ac8500d391ea212ac2a139ca06319f731314c78155e9e42caf9" Nov 03 22:45:07.803626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.803602 4903 scope.go:122] "RemoveContainer" containerID="136fbabc17d6a9ba3fa043de1dcffd7aa8e4d772f7564c96de6b7d236d3fe731" Nov 03 22:45:07.819029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.818863 4903 scope.go:122] "RemoveContainer" containerID="a44311d6b9ada1d25d9db52536707c8f69a66c06b3661266a5accaa5f0a35ea0" Nov 03 22:45:07.834854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.834827 4903 scope.go:122] "RemoveContainer" containerID="40abb25d43764f97ea2930c68598a64c312122d8503f21542e0f0dc90a114382" Nov 03 22:45:07.848519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.848495 4903 scope.go:122] "RemoveContainer" containerID="4a3503f3e5ae885cd20e57d5cb9c00677d55142e38601e3f34b3af05025a2873" Nov 03 22:45:07.862261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:07.862239 4903 scope.go:122] "RemoveContainer" containerID="6d1f374375ff1d4e81f6fb1f612c64e65b39811ea7909908730e43ad88a76727" Nov 03 22:45:08.098895 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.098857 4903 scope.go:122] "RemoveContainer" containerID="4b98ad0cb672aba4321624a9249554ddc3d995c2f6eb689bc7c2e98f120d1aba" Nov 03 22:45:08.099451 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.099336 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr_odh-ai-gateway-infra(b10ff0e7-d241-428a-9e02-ac902f55b8c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" Nov 03 22:45:08.100649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.100622 4903 scope.go:122] "RemoveContainer" containerID="5427e22b12520acb7a80a3dc26c037e893eef49ea254b17ffc84618fa971c2e1" Nov 03 22:45:08.101024 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.100999 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj_odh-ai-gateway-infra(663b664e-98af-4583-aed0-da9572e83f52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" podUID="663b664e-98af-4583-aed0-da9572e83f52" Nov 03 22:45:08.102336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.102285 4903 scope.go:122] "RemoveContainer" containerID="3040426ac54dd251b5e5ca081b54b45d23c3a2d6a107f4ae6e46f77621491ec3" Nov 03 22:45:08.102698 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.102675 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s_odh-ai-gateway-infra(659a4a6e-2d77-4d9f-8284-a29be2d2de56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" Nov 03 22:45:08.104009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.103984 4903 scope.go:122] "RemoveContainer" containerID="b9853b451bc09b7a0b9d47a6bd5a432e8114df42f87d4a60f6814dc706bcf7c1" Nov 03 22:45:08.104305 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.104270 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw_odh-ai-gateway-infra(d177e30a-9ef2-4d4d-a2be-3872a6bc9411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" Nov 03 22:45:08.105644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.105625 4903 scope.go:122] "RemoveContainer" containerID="b88f4818986ec455b0d0a9cd309803b0f0ae97aef0607b5351a0fe60f77405bf" Nov 03 22:45:08.105920 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.105902 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp_odh-ai-gateway-infra(85615752-db5a-4dea-bc86-290e793928e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" podUID="85615752-db5a-4dea-bc86-290e793928e0" Nov 03 22:45:08.107401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.107367 4903 scope.go:122] "RemoveContainer" containerID="b5cd8682f0f58c75bf1d25530ab6f266dee97cb2283859bd0eba1eff81230dff" Nov 03 22:45:08.107711 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.107686 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4_odh-ai-gateway-infra(b052a69f-e1a3-408e-9881-94770cb92ed6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" Nov 03 22:45:08.108459 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.108429 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:08.109212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.109196 4903 scope.go:122] "RemoveContainer" containerID="86434e39ef5458fe446d09f9df003b6091404d556697220e2df4c6b8992fe064" Nov 03 22:45:08.109485 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.109459 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh_odh-ai-gateway-infra(7c71cca5-adfa-4f08-9233-2719cadc9550)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" Nov 03 22:45:08.110950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.110931 4903 scope.go:122] "RemoveContainer" containerID="b48831d65fc923bbcbb377a8026c1d6eac310556cbfcbfac414e94b495314c39" Nov 03 22:45:08.111184 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.111154 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv_odh-ai-gateway-infra(1010f884-c3b1-442b-bd6d-848666e3bfb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" Nov 03 22:45:08.112846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.112793 4903 scope.go:122] "RemoveContainer" containerID="53d7e5166594cdc40ff7ccee53cb6e6ac458a4645884f5079bc8e33dfefac01d" Nov 03 22:45:08.113065 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.113029 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p_odh-ai-gateway-infra(fb962f11-d246-41e8-a596-6383a2d91a58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" Nov 03 22:45:08.114468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.114448 4903 generic.go:356] "Generic (PLEG): container finished" podID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerID="afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b" exitCode=0 Nov 03 22:45:08.114574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.114524 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" event={"ID":"eb281e27-8506-4de9-91d3-4800a8f40dc3","Type":"ContainerDied","Data":"afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b"} Nov 03 22:45:08.114932 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.114916 4903 scope.go:122] "RemoveContainer" containerID="afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b" Nov 03 22:45:08.116719 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.116702 4903 scope.go:122] "RemoveContainer" containerID="a671baa0aa3d368f211fcbff25b2bd1345b1e47d8732a99fec4b8af6888966b5" Nov 03 22:45:08.116989 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.116961 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d_odh-ai-gateway-infra(598378e1-2604-411f-9b3c-e2a196b4dc2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" Nov 03 22:45:08.117117 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.117005 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:08.117117 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.117085 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:08.117241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.117130 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:08.118989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.118821 4903 scope.go:122] "RemoveContainer" containerID="028c09f8ff7f38e8b0a631ea28f8ab00a8f8e29485b0409f8b8e5b46edce0d28" Nov 03 22:45:08.119838 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.119816 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q_odh-ai-gateway-infra(3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" Nov 03 22:45:08.120521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.120505 4903 scope.go:122] "RemoveContainer" containerID="e4c35972457370c4818106312d3b47db41ad42facfb2584e8a365f6964a23288" Nov 03 22:45:08.120708 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.120690 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5_odh-ai-gateway-infra(9445c683-e8c3-41bf-9cbc-5d3093d840c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" Nov 03 22:45:08.122589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.122573 4903 scope.go:122] "RemoveContainer" containerID="9ef92eaa7acdd3a7b4ec388df510294a2db2296620866929e5d7f46296e22ee3" Nov 03 22:45:08.122836 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.122814 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps_odh-ai-gateway-infra(3ec653a6-42f1-49b3-a7d9-827945ef8946)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" Nov 03 22:45:08.124372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.124351 4903 scope.go:122] "RemoveContainer" containerID="c05195136efc5519a63d72bd375cf87a40804e9b45707df7c5470ddbdb776541" Nov 03 22:45:08.124577 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.124557 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn_odh-ai-gateway-infra(2f271965-1d01-4f06-8619-ac80aaeb2154)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" Nov 03 22:45:08.126209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.126194 4903 scope.go:122] "RemoveContainer" containerID="d8726ab9b34a419f4463dbafeccff9865592c62119cab7df872452fab0707ed5" Nov 03 22:45:08.126434 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.126414 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq_odh-ai-gateway-infra(1b686411-6854-4f2a-9400-8632d8e55009)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" podUID="1b686411-6854-4f2a-9400-8632d8e55009" Nov 03 22:45:08.127957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.127938 4903 scope.go:122] "RemoveContainer" containerID="43358d3bc9debb1679337cf29be821b4a2bc01f496dd7d01317366a582ff7380" Nov 03 22:45:08.128188 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.128168 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r_odh-ai-gateway-infra(31ad238a-db65-4160-a25b-9e29df9dbc94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" Nov 03 22:45:08.129750 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.129734 4903 scope.go:122] "RemoveContainer" containerID="61f9aab06d7e39cc3ebabeab31aa3e47c05b6617a55c9e1fa04a450280e1a2fc" Nov 03 22:45:08.129979 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.129955 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62_odh-ai-gateway-infra(4e3fdb74-d353-4007-a96b-3af3db01cc57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" Nov 03 22:45:08.131468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.131451 4903 scope.go:122] "RemoveContainer" containerID="5ac015103701def7a9eac439e282a47193b9adf15b9e6ed97edd79af5cffba83" Nov 03 22:45:08.131710 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.131686 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg_odh-ai-gateway-infra(c9d6c159-15e1-4fc7-8d50-40758bd126be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" Nov 03 22:45:08.133072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.133054 4903 scope.go:122] "RemoveContainer" containerID="70b5fd2693aa61533c4e7e588e490cdef597a668f19a6626bdb9aaef6fc5b078" Nov 03 22:45:08.133327 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.133305 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc_odh-ai-gateway-infra(1d9725d5-d50c-496b-834e-a783e0c47b28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" Nov 03 22:45:08.134727 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.134697 4903 scope.go:122] "RemoveContainer" containerID="78a5d306823b20bc8a2ece7fda9266f8ad1651d2a90449123993ba583b1912d2" Nov 03 22:45:08.134941 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.134916 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p_odh-ai-gateway-infra(c63f0f29-03cf-4880-8e34-b44f25a1eb2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" Nov 03 22:45:08.147367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:08.147346 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:45:08.807128 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:08.807095 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:09.168324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:09.166049 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" event={"ID":"eb281e27-8506-4de9-91d3-4800a8f40dc3","Type":"ContainerStarted","Data":"9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4"} Nov 03 22:45:09.935791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:09.935753 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:45:09.937022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:09.936999 4903 patch_prober.go:29] interesting pod/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 22:45:09.937149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:09.937053 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 22:45:10.935427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:10.935394 4903 patch_prober.go:29] interesting pod/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 22:45:10.935813 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:10.935445 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 22:45:11.935489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:11.935450 4903 patch_prober.go:29] interesting pod/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 22:45:11.935963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:11.935511 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 22:45:12.403585 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:12.403540 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:12.404222 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:12.404190 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:12.935645 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:12.935602 4903 patch_prober.go:29] interesting pod/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 22:45:12.936056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:12.935663 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 22:45:13.935940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:13.935900 4903 patch_prober.go:29] interesting pod/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 22:45:13.936426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:13.935963 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 22:45:14.522377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.522339 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq"] Nov 03 22:45:14.523523 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.523504 4903 cpu_manager.go:414] "containerMap: removing container" podUID="40bf741a-dbda-425a-94b3-27e46a7d1a94" containerName="main" Nov 03 22:45:14.523589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.523525 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bf741a-dbda-425a-94b3-27e46a7d1a94" containerName="main" Nov 03 22:45:14.523712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.523703 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bf741a-dbda-425a-94b3-27e46a7d1a94" containerName="main" Nov 03 22:45:14.527612 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.527595 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.560909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.560874 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7940ea-openshift-default-dockercfg-58cvs\"" Nov 03 22:45:14.562828 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.562801 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq"] Nov 03 22:45:14.588105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.588060 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.588353 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.588157 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-envoy\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.588353 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.588192 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-data\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.588353 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.588244 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-podinfo\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.588557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.588421 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlg6\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-kube-api-access-6mlg6\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.588557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.588497 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-credential-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.588557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.588540 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-token\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.588679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.588571 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istiod-ca-cert\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.588679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.588612 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-certs\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690165 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlg6\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-kube-api-access-6mlg6\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690268 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-credential-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690362 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-token\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690406 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istiod-ca-cert\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690467 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-certs\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690551 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690633 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-envoy\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690753 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-credential-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690779 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-data\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690851 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-certs\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.690918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690884 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-podinfo\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.691021 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.690986 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-socket\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.691108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.691085 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-data\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.691266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.691236 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istiod-ca-cert\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.693486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.693463 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-envoy\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.693698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.693676 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-podinfo\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.711857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.711829 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-token\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.712006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.711945 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlg6\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-kube-api-access-6mlg6\") pod \"e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.857267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.857178 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:14.944322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.944269 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:45:14.946335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.945786 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:45:14.946459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:14.946337 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:45:15.193881 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:15.193852 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda619613c_1ab5_4af3_b687_dd4a9a3f6abb.slice/crio-e261abf5dec49645f2b33a3be76b204137169a63b46d8f6f9377cb60741b6ff7 WatchSource:0}: Error finding container e261abf5dec49645f2b33a3be76b204137169a63b46d8f6f9377cb60741b6ff7: Status 404 returned error can't find the container with id e261abf5dec49645f2b33a3be76b204137169a63b46d8f6f9377cb60741b6ff7 Nov 03 22:45:15.194180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:15.194151 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq"] Nov 03 22:45:15.197612 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:15.197580 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:15.197728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:15.197644 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:15.197728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:15.197673 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:45:15.238389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:15.237101 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" event={"ID":"a619613c-1ab5-4af3-b687-dd4a9a3f6abb","Type":"ContainerStarted","Data":"e261abf5dec49645f2b33a3be76b204137169a63b46d8f6f9377cb60741b6ff7"} Nov 03 22:45:16.272003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:16.271963 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" event={"ID":"a619613c-1ab5-4af3-b687-dd4a9a3f6abb","Type":"ContainerStarted","Data":"0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545"} Nov 03 22:45:16.322002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:16.321954 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" podStartSLOduration=2.3219411770000002 podStartE2EDuration="2.321941177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:16.316958386 +0000 UTC m=+1942.720899386" watchObservedRunningTime="2026-11-03 22:45:16.321941177 +0000 UTC m=+1942.725882152" Nov 03 22:45:16.725986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:16.725958 4903 scope.go:122] "RemoveContainer" containerID="d8726ab9b34a419f4463dbafeccff9865592c62119cab7df872452fab0707ed5" Nov 03 22:45:16.726176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:16.726065 4903 scope.go:122] "RemoveContainer" containerID="70b5fd2693aa61533c4e7e588e490cdef597a668f19a6626bdb9aaef6fc5b078" Nov 03 22:45:16.861384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:16.861352 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:16.865845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:16.865447 4903 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 22:45:16.865845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:16.865518 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 22:45:17.725914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:17.725879 4903 scope.go:122] "RemoveContainer" containerID="78a5d306823b20bc8a2ece7fda9266f8ad1651d2a90449123993ba583b1912d2" Nov 03 22:45:17.726546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:17.726515 4903 scope.go:122] "RemoveContainer" containerID="e4c35972457370c4818106312d3b47db41ad42facfb2584e8a365f6964a23288" Nov 03 22:45:17.727459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:17.726745 4903 scope.go:122] "RemoveContainer" containerID="c05195136efc5519a63d72bd375cf87a40804e9b45707df7c5470ddbdb776541" Nov 03 22:45:17.861321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:17.861007 4903 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 22:45:17.861491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:17.861241 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 22:45:18.303561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.303320 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" event={"ID":"1b686411-6854-4f2a-9400-8632d8e55009","Type":"ContainerDied","Data":"d40b082313ea61848721c79c9ba594a74e28dde846fabb44e5c23744abd57a31"} Nov 03 22:45:18.303561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.303404 4903 scope.go:122] "RemoveContainer" containerID="d8726ab9b34a419f4463dbafeccff9865592c62119cab7df872452fab0707ed5" Nov 03 22:45:18.304464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.303830 4903 scope.go:122] "RemoveContainer" containerID="d40b082313ea61848721c79c9ba594a74e28dde846fabb44e5c23744abd57a31" Nov 03 22:45:18.304464 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:18.304041 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:18.304464 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:18.304237 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq_odh-ai-gateway-infra(1b686411-6854-4f2a-9400-8632d8e55009)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" podUID="1b686411-6854-4f2a-9400-8632d8e55009" Nov 03 22:45:18.304464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.304346 4903 generic.go:356] "Generic (PLEG): container finished" podID="1b686411-6854-4f2a-9400-8632d8e55009" containerID="d40b082313ea61848721c79c9ba594a74e28dde846fabb44e5c23744abd57a31" exitCode=60 Nov 03 22:45:18.306921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.306897 4903 generic.go:356] "Generic (PLEG): container finished" podID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerID="d7eb7a3f078af088a1d7bd444814b62b5ab803dabdb8d3f81e133b246826d561" exitCode=60 Nov 03 22:45:18.307038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.306973 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" event={"ID":"1d9725d5-d50c-496b-834e-a783e0c47b28","Type":"ContainerDied","Data":"d7eb7a3f078af088a1d7bd444814b62b5ab803dabdb8d3f81e133b246826d561"} Nov 03 22:45:18.307420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.307403 4903 scope.go:122] "RemoveContainer" containerID="d7eb7a3f078af088a1d7bd444814b62b5ab803dabdb8d3f81e133b246826d561" Nov 03 22:45:18.307812 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:18.307791 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc_odh-ai-gateway-infra(1d9725d5-d50c-496b-834e-a783e0c47b28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" Nov 03 22:45:18.322853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.322834 4903 scope.go:122] "RemoveContainer" containerID="70b5fd2693aa61533c4e7e588e490cdef597a668f19a6626bdb9aaef6fc5b078" Nov 03 22:45:18.407489 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:18.407448 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:45:18.725647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.725618 4903 scope.go:122] "RemoveContainer" containerID="a671baa0aa3d368f211fcbff25b2bd1345b1e47d8732a99fec4b8af6888966b5" Nov 03 22:45:18.725849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.725753 4903 scope.go:122] "RemoveContainer" containerID="028c09f8ff7f38e8b0a631ea28f8ab00a8f8e29485b0409f8b8e5b46edce0d28" Nov 03 22:45:18.726276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.726255 4903 scope.go:122] "RemoveContainer" containerID="53d7e5166594cdc40ff7ccee53cb6e6ac458a4645884f5079bc8e33dfefac01d" Nov 03 22:45:18.727098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.727074 4903 scope.go:122] "RemoveContainer" containerID="b9853b451bc09b7a0b9d47a6bd5a432e8114df42f87d4a60f6814dc706bcf7c1" Nov 03 22:45:18.859649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.859190 4903 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 22:45:18.859649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:18.859247 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 22:45:19.327209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.327173 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" event={"ID":"fb962f11-d246-41e8-a596-6383a2d91a58","Type":"ContainerStarted","Data":"61287a576885fa52da1bf1ee3fa979ebe0be10405172183633d61553ba2abedf"} Nov 03 22:45:19.329764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.329737 4903 generic.go:356] "Generic (PLEG): container finished" podID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerID="361953a8e763ad902f87ffe83d45c439006f20dfd98de65e0638566135101f75" exitCode=60 Nov 03 22:45:19.329913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.329822 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" event={"ID":"9445c683-e8c3-41bf-9cbc-5d3093d840c6","Type":"ContainerDied","Data":"361953a8e763ad902f87ffe83d45c439006f20dfd98de65e0638566135101f75"} Nov 03 22:45:19.329913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.329869 4903 scope.go:122] "RemoveContainer" containerID="e4c35972457370c4818106312d3b47db41ad42facfb2584e8a365f6964a23288" Nov 03 22:45:19.330619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.330269 4903 scope.go:122] "RemoveContainer" containerID="361953a8e763ad902f87ffe83d45c439006f20dfd98de65e0638566135101f75" Nov 03 22:45:19.330901 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:19.330879 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5_odh-ai-gateway-infra(9445c683-e8c3-41bf-9cbc-5d3093d840c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" Nov 03 22:45:19.333432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.333413 4903 generic.go:356] "Generic (PLEG): container finished" podID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerID="aa131a0d93080197515072f519134ff833e3f49a3124b28d814cc5f7d46baa0f" exitCode=60 Nov 03 22:45:19.333508 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.333484 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" event={"ID":"2f271965-1d01-4f06-8619-ac80aaeb2154","Type":"ContainerDied","Data":"aa131a0d93080197515072f519134ff833e3f49a3124b28d814cc5f7d46baa0f"} Nov 03 22:45:19.333741 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.333727 4903 scope.go:122] "RemoveContainer" containerID="aa131a0d93080197515072f519134ff833e3f49a3124b28d814cc5f7d46baa0f" Nov 03 22:45:19.333954 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:19.333934 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn_odh-ai-gateway-infra(2f271965-1d01-4f06-8619-ac80aaeb2154)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" Nov 03 22:45:19.339853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.339834 4903 generic.go:356] "Generic (PLEG): container finished" podID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerID="86a8a3f6be964e4f93ec1c674accebadbf6860cfe792083efdbf4699cf7261a8" exitCode=60 Nov 03 22:45:19.339973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.339962 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" event={"ID":"c63f0f29-03cf-4880-8e34-b44f25a1eb2a","Type":"ContainerDied","Data":"86a8a3f6be964e4f93ec1c674accebadbf6860cfe792083efdbf4699cf7261a8"} Nov 03 22:45:19.340226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.340214 4903 scope.go:122] "RemoveContainer" containerID="86a8a3f6be964e4f93ec1c674accebadbf6860cfe792083efdbf4699cf7261a8" Nov 03 22:45:19.363314 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:19.340491 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p_odh-ai-gateway-infra(c63f0f29-03cf-4880-8e34-b44f25a1eb2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" Nov 03 22:45:19.376187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.376051 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq"] Nov 03 22:45:19.382597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.382578 4903 scope.go:122] "RemoveContainer" containerID="c05195136efc5519a63d72bd375cf87a40804e9b45707df7c5470ddbdb776541" Nov 03 22:45:19.390831 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.390782 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" podStartSLOduration=19.390764395 podStartE2EDuration="19.390764395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:19.387815913 +0000 UTC m=+1945.791756889" watchObservedRunningTime="2026-11-03 22:45:19.390764395 +0000 UTC m=+1945.794705372" Nov 03 22:45:19.398695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.398660 4903 scope.go:122] "RemoveContainer" containerID="78a5d306823b20bc8a2ece7fda9266f8ad1651d2a90449123993ba583b1912d2" Nov 03 22:45:19.445458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.441195 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc"] Nov 03 22:45:19.729968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.725745 4903 scope.go:122] "RemoveContainer" containerID="b48831d65fc923bbcbb377a8026c1d6eac310556cbfcbfac414e94b495314c39" Nov 03 22:45:19.729968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.727483 4903 scope.go:122] "RemoveContainer" containerID="5427e22b12520acb7a80a3dc26c037e893eef49ea254b17ffc84618fa971c2e1" Nov 03 22:45:19.729968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.728184 4903 scope.go:122] "RemoveContainer" containerID="9ef92eaa7acdd3a7b4ec388df510294a2db2296620866929e5d7f46296e22ee3" Nov 03 22:45:19.729968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.728619 4903 scope.go:122] "RemoveContainer" containerID="5ac015103701def7a9eac439e282a47193b9adf15b9e6ed97edd79af5cffba83" Nov 03 22:45:19.729968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.729361 4903 scope.go:122] "RemoveContainer" containerID="b5cd8682f0f58c75bf1d25530ab6f266dee97cb2283859bd0eba1eff81230dff" Nov 03 22:45:19.803443 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.802515 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:19.816609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.816580 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x"] Nov 03 22:45:19.818208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.818180 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1b686411-6854-4f2a-9400-8632d8e55009" containerName="cleanup" Nov 03 22:45:19.818366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.818210 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b686411-6854-4f2a-9400-8632d8e55009" containerName="cleanup" Nov 03 22:45:19.818366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.818265 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1b686411-6854-4f2a-9400-8632d8e55009" containerName="cleanup" Nov 03 22:45:19.818366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.818323 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b686411-6854-4f2a-9400-8632d8e55009" containerName="cleanup" Nov 03 22:45:19.818366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.818351 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1b686411-6854-4f2a-9400-8632d8e55009" containerName="cleanup" Nov 03 22:45:19.818366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.818361 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b686411-6854-4f2a-9400-8632d8e55009" containerName="cleanup" Nov 03 22:45:19.818706 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.818687 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b686411-6854-4f2a-9400-8632d8e55009" containerName="cleanup" Nov 03 22:45:19.818771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.818734 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b686411-6854-4f2a-9400-8632d8e55009" containerName="cleanup" Nov 03 22:45:19.881329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.880423 4903 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 22:45:19.881329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.880485 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 22:45:19.882480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.882455 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.888373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.888349 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x"] Nov 03 22:45:19.894928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.894903 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-tmp-dir\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.895117 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.895096 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-model-cache\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.895274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.895255 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-home\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.895776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.895752 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-dshm\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.895959 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.895937 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17306d5-2b3b-4d30-9a47-14920036a2ef-tls-certs\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.896501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.896486 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dpj\" (UniqueName: \"kubernetes.io/projected/b17306d5-2b3b-4d30-9a47-14920036a2ef-kube-api-access-r5dpj\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.907025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.906453 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-963936\"/\"kube-root-ca.crt\"" Nov 03 22:45:19.907025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.906536 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-963936\"/\"openshift-service-ca.crt\"" Nov 03 22:45:19.907025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.906625 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-963936\"/\"default-dockercfg-ml9pc\"" Nov 03 22:45:19.907025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.906687 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-963936\"/\"test-model-963936-kserve-self-signed-certs\"" Nov 03 22:45:19.909108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.909086 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:19.998254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.997519 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b686411-6854-4f2a-9400-8632d8e55009-kube-api-access-g66h6\" (UniqueName: \"kubernetes.io/projected/1b686411-6854-4f2a-9400-8632d8e55009-kube-api-access-g66h6\") pod \"1b686411-6854-4f2a-9400-8632d8e55009\" (UID: \"1b686411-6854-4f2a-9400-8632d8e55009\") " Nov 03 22:45:19.998254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.997604 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9725d5-d50c-496b-834e-a783e0c47b28-kube-api-access-ljk2w\" (UniqueName: \"kubernetes.io/projected/1d9725d5-d50c-496b-834e-a783e0c47b28-kube-api-access-ljk2w\") pod \"1d9725d5-d50c-496b-834e-a783e0c47b28\" (UID: \"1d9725d5-d50c-496b-834e-a783e0c47b28\") " Nov 03 22:45:19.998254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.997629 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9725d5-d50c-496b-834e-a783e0c47b28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9725d5-d50c-496b-834e-a783e0c47b28-maas-api-service-ca\") pod \"1d9725d5-d50c-496b-834e-a783e0c47b28\" (UID: \"1d9725d5-d50c-496b-834e-a783e0c47b28\") " Nov 03 22:45:19.998254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.997712 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b686411-6854-4f2a-9400-8632d8e55009-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b686411-6854-4f2a-9400-8632d8e55009-maas-api-service-ca\") pod \"1b686411-6854-4f2a-9400-8632d8e55009\" (UID: \"1b686411-6854-4f2a-9400-8632d8e55009\") " Nov 03 22:45:19.999446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.998623 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-dshm\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.999446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.998668 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17306d5-2b3b-4d30-9a47-14920036a2ef-tls-certs\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.999446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.998708 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dpj\" (UniqueName: \"kubernetes.io/projected/b17306d5-2b3b-4d30-9a47-14920036a2ef-kube-api-access-r5dpj\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.999446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.998786 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-tmp-dir\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.999446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.998838 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-model-cache\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.999446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.998902 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-home\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:19.999906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:19.999746 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-home\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:20.000351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.000079 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9725d5-d50c-496b-834e-a783e0c47b28-maas-api-service-ca" pod "1d9725d5-d50c-496b-834e-a783e0c47b28" (UID: "1d9725d5-d50c-496b-834e-a783e0c47b28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:20.000920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.000612 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b686411-6854-4f2a-9400-8632d8e55009-maas-api-service-ca" pod "1b686411-6854-4f2a-9400-8632d8e55009" (UID: "1b686411-6854-4f2a-9400-8632d8e55009"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:20.001164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.001074 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-tmp-dir\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:20.001354 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.001331 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-model-cache\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:20.004329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.003726 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17306d5-2b3b-4d30-9a47-14920036a2ef-tls-certs\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:20.004800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.004781 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-dshm\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:20.022684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.022632 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dpj\" (UniqueName: \"kubernetes.io/projected/b17306d5-2b3b-4d30-9a47-14920036a2ef-kube-api-access-r5dpj\") pod \"test-model-963936-kserve-5fdcd9bc5f-cqj4x\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:20.093499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.084494 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9725d5-d50c-496b-834e-a783e0c47b28-kube-api-access-ljk2w" pod "1d9725d5-d50c-496b-834e-a783e0c47b28" (UID: "1d9725d5-d50c-496b-834e-a783e0c47b28"). InnerVolumeSpecName "kube-api-access-ljk2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:20.123364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.098529 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b686411-6854-4f2a-9400-8632d8e55009-kube-api-access-g66h6" pod "1b686411-6854-4f2a-9400-8632d8e55009" (UID: "1b686411-6854-4f2a-9400-8632d8e55009"). InnerVolumeSpecName "kube-api-access-g66h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:20.124133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.114443 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b686411-6854-4f2a-9400-8632d8e55009-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:20.137507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.124337 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljk2w\" (UniqueName: \"kubernetes.io/projected/1d9725d5-d50c-496b-834e-a783e0c47b28-kube-api-access-ljk2w\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:20.137785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.137739 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d9725d5-d50c-496b-834e-a783e0c47b28-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:20.287839 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.264904 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g66h6\" (UniqueName: \"kubernetes.io/projected/1b686411-6854-4f2a-9400-8632d8e55009-kube-api-access-g66h6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:20.311515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.310582 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:20.382413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.382377 4903 generic.go:356] "Generic (PLEG): container finished" podID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerID="fe274ea4319ec218295251e145721df46e121c0a24a9d319190e26e1d7dd1b88" exitCode=60 Nov 03 22:45:20.382627 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.382533 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" event={"ID":"d177e30a-9ef2-4d4d-a2be-3872a6bc9411","Type":"ContainerDied","Data":"fe274ea4319ec218295251e145721df46e121c0a24a9d319190e26e1d7dd1b88"} Nov 03 22:45:20.382627 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.382577 4903 scope.go:122] "RemoveContainer" containerID="b9853b451bc09b7a0b9d47a6bd5a432e8114df42f87d4a60f6814dc706bcf7c1" Nov 03 22:45:20.383015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.382995 4903 scope.go:122] "RemoveContainer" containerID="fe274ea4319ec218295251e145721df46e121c0a24a9d319190e26e1d7dd1b88" Nov 03 22:45:20.383403 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:20.383374 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw_odh-ai-gateway-infra(d177e30a-9ef2-4d4d-a2be-3872a6bc9411)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" Nov 03 22:45:20.385403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.385369 4903 generic.go:356] "Generic (PLEG): container finished" podID="fb962f11-d246-41e8-a596-6383a2d91a58" containerID="61287a576885fa52da1bf1ee3fa979ebe0be10405172183633d61553ba2abedf" exitCode=60 Nov 03 22:45:20.385511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.385399 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" event={"ID":"fb962f11-d246-41e8-a596-6383a2d91a58","Type":"ContainerDied","Data":"61287a576885fa52da1bf1ee3fa979ebe0be10405172183633d61553ba2abedf"} Nov 03 22:45:20.385858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.385837 4903 scope.go:122] "RemoveContainer" containerID="61287a576885fa52da1bf1ee3fa979ebe0be10405172183633d61553ba2abedf" Nov 03 22:45:20.386218 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:20.386196 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p_odh-ai-gateway-infra(fb962f11-d246-41e8-a596-6383a2d91a58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" Nov 03 22:45:20.387605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.387572 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" event={"ID":"598378e1-2604-411f-9b3c-e2a196b4dc2e","Type":"ContainerStarted","Data":"7aa8df1636df61a58b2b2cc344da2a19a07f41fc623c7f277e726f4abec36876"} Nov 03 22:45:20.389489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.389469 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" event={"ID":"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164","Type":"ContainerStarted","Data":"d7a07a5a3e24739f9c8fd6542ba6cc8ad6740f31728d55661181c8116e12c1dd"} Nov 03 22:45:20.393858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.393840 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" event={"ID":"1b686411-6854-4f2a-9400-8632d8e55009","Type":"ContainerDied","Data":"db475a8a533fb12d85049512e6e887b9f76a54e90dde26934c6484db86a2ffc3"} Nov 03 22:45:20.393933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.393921 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq" Nov 03 22:45:20.396921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.396899 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" event={"ID":"1d9725d5-d50c-496b-834e-a783e0c47b28","Type":"ContainerDied","Data":"ea13a1e0d941a86fa48eadb648e6fcc7affbcdc55a77edfa3a59320f9be8a3be"} Nov 03 22:45:20.397940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.397925 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc" Nov 03 22:45:20.402053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.402033 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p"] Nov 03 22:45:20.489256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.489222 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn"] Nov 03 22:45:20.527692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.527667 4903 scope.go:122] "RemoveContainer" containerID="53d7e5166594cdc40ff7ccee53cb6e6ac458a4645884f5079bc8e33dfefac01d" Nov 03 22:45:20.535213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.534898 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p"] Nov 03 22:45:20.599276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.597009 4903 scope.go:122] "RemoveContainer" containerID="d40b082313ea61848721c79c9ba594a74e28dde846fabb44e5c23744abd57a31" Nov 03 22:45:20.617326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.617249 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq"] Nov 03 22:45:20.639559 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.639516 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895765-62rkq"] Nov 03 22:45:20.672556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.672528 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5"] Nov 03 22:45:20.674659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.674640 4903 scope.go:122] "RemoveContainer" containerID="d7eb7a3f078af088a1d7bd444814b62b5ab803dabdb8d3f81e133b246826d561" Nov 03 22:45:20.691617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.691528 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc"] Nov 03 22:45:20.702163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.702133 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-124490-29895765-5v7sc"] Nov 03 22:45:20.727040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.726632 4903 scope.go:122] "RemoveContainer" containerID="61f9aab06d7e39cc3ebabeab31aa3e47c05b6617a55c9e1fa04a450280e1a2fc" Nov 03 22:45:20.727901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.727279 4903 scope.go:122] "RemoveContainer" containerID="3040426ac54dd251b5e5ca081b54b45d23c3a2d6a107f4ae6e46f77621491ec3" Nov 03 22:45:20.727901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.727445 4903 scope.go:122] "RemoveContainer" containerID="43358d3bc9debb1679337cf29be821b4a2bc01f496dd7d01317366a582ff7380" Nov 03 22:45:20.727901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.727832 4903 scope.go:122] "RemoveContainer" containerID="4b98ad0cb672aba4321624a9249554ddc3d995c2f6eb689bc7c2e98f120d1aba" Nov 03 22:45:20.728303 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.728269 4903 scope.go:122] "RemoveContainer" containerID="b88f4818986ec455b0d0a9cd309803b0f0ae97aef0607b5351a0fe60f77405bf" Nov 03 22:45:20.728914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.728514 4903 scope.go:122] "RemoveContainer" containerID="86434e39ef5458fe446d09f9df003b6091404d556697220e2df4c6b8992fe064" Nov 03 22:45:20.875371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.872481 4903 patch_prober.go:29] interesting pod/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 22:45:20.875371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.872536 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 22:45:20.998335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:20.995701 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x"] Nov 03 22:45:21.216615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.216595 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:21.225430 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.225409 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:21.234318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.233788 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:21.324739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.323944 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9445c683-e8c3-41bf-9cbc-5d3093d840c6-kube-api-access-5g7h9\" (UniqueName: \"kubernetes.io/projected/9445c683-e8c3-41bf-9cbc-5d3093d840c6-kube-api-access-5g7h9\") pod \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\" (UID: \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\") " Nov 03 22:45:21.324739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.324068 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-kube-api-access-pqjgh\" (UniqueName: \"kubernetes.io/projected/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-kube-api-access-pqjgh\") pod \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\" (UID: \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\") " Nov 03 22:45:21.324739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.324165 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-maas-api-service-ca\") pod \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\" (UID: \"c63f0f29-03cf-4880-8e34-b44f25a1eb2a\") " Nov 03 22:45:21.324739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.324186 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f271965-1d01-4f06-8619-ac80aaeb2154-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f271965-1d01-4f06-8619-ac80aaeb2154-maas-api-service-ca\") pod \"2f271965-1d01-4f06-8619-ac80aaeb2154\" (UID: \"2f271965-1d01-4f06-8619-ac80aaeb2154\") " Nov 03 22:45:21.324739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.324345 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9445c683-e8c3-41bf-9cbc-5d3093d840c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445c683-e8c3-41bf-9cbc-5d3093d840c6-maas-api-service-ca\") pod \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\" (UID: \"9445c683-e8c3-41bf-9cbc-5d3093d840c6\") " Nov 03 22:45:21.324739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.324532 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f271965-1d01-4f06-8619-ac80aaeb2154-kube-api-access-7gvt8\" (UniqueName: \"kubernetes.io/projected/2f271965-1d01-4f06-8619-ac80aaeb2154-kube-api-access-7gvt8\") pod \"2f271965-1d01-4f06-8619-ac80aaeb2154\" (UID: \"2f271965-1d01-4f06-8619-ac80aaeb2154\") " Nov 03 22:45:21.325086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.324898 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9445c683-e8c3-41bf-9cbc-5d3093d840c6-maas-api-service-ca" pod "9445c683-e8c3-41bf-9cbc-5d3093d840c6" (UID: "9445c683-e8c3-41bf-9cbc-5d3093d840c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:21.325820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.325784 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9445c683-e8c3-41bf-9cbc-5d3093d840c6-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:21.326149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.326124 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f271965-1d01-4f06-8619-ac80aaeb2154-maas-api-service-ca" pod "2f271965-1d01-4f06-8619-ac80aaeb2154" (UID: "2f271965-1d01-4f06-8619-ac80aaeb2154"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:21.326820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.326785 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-maas-api-service-ca" pod "c63f0f29-03cf-4880-8e34-b44f25a1eb2a" (UID: "c63f0f29-03cf-4880-8e34-b44f25a1eb2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:21.354844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.354558 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f271965-1d01-4f06-8619-ac80aaeb2154-kube-api-access-7gvt8" pod "2f271965-1d01-4f06-8619-ac80aaeb2154" (UID: "2f271965-1d01-4f06-8619-ac80aaeb2154"). InnerVolumeSpecName "kube-api-access-7gvt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:21.354844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.354699 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9445c683-e8c3-41bf-9cbc-5d3093d840c6-kube-api-access-5g7h9" pod "9445c683-e8c3-41bf-9cbc-5d3093d840c6" (UID: "9445c683-e8c3-41bf-9cbc-5d3093d840c6"). InnerVolumeSpecName "kube-api-access-5g7h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:21.354844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.354806 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-kube-api-access-pqjgh" pod "c63f0f29-03cf-4880-8e34-b44f25a1eb2a" (UID: "c63f0f29-03cf-4880-8e34-b44f25a1eb2a"). InnerVolumeSpecName "kube-api-access-pqjgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:21.458032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.451186 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gvt8\" (UniqueName: \"kubernetes.io/projected/2f271965-1d01-4f06-8619-ac80aaeb2154-kube-api-access-7gvt8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:21.458032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.451224 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g7h9\" (UniqueName: \"kubernetes.io/projected/9445c683-e8c3-41bf-9cbc-5d3093d840c6-kube-api-access-5g7h9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:21.458032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.451242 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqjgh\" (UniqueName: \"kubernetes.io/projected/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-kube-api-access-pqjgh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:21.458032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.451260 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63f0f29-03cf-4880-8e34-b44f25a1eb2a-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:21.458032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.451274 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f271965-1d01-4f06-8619-ac80aaeb2154-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:21.474888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.459658 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw"] Nov 03 22:45:21.488434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.487005 4903 generic.go:356] "Generic (PLEG): container finished" podID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerID="7aa8df1636df61a58b2b2cc344da2a19a07f41fc623c7f277e726f4abec36876" exitCode=60 Nov 03 22:45:21.488434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.487116 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" event={"ID":"598378e1-2604-411f-9b3c-e2a196b4dc2e","Type":"ContainerDied","Data":"7aa8df1636df61a58b2b2cc344da2a19a07f41fc623c7f277e726f4abec36876"} Nov 03 22:45:21.488434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.487173 4903 scope.go:122] "RemoveContainer" containerID="a671baa0aa3d368f211fcbff25b2bd1345b1e47d8732a99fec4b8af6888966b5" Nov 03 22:45:21.488434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.487724 4903 scope.go:122] "RemoveContainer" containerID="7aa8df1636df61a58b2b2cc344da2a19a07f41fc623c7f277e726f4abec36876" Nov 03 22:45:21.488434 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:21.488217 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d_odh-ai-gateway-infra(598378e1-2604-411f-9b3c-e2a196b4dc2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" Nov 03 22:45:21.492024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.491995 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q"] Nov 03 22:45:21.498284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.498256 4903 generic.go:356] "Generic (PLEG): container finished" podID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerID="d7a07a5a3e24739f9c8fd6542ba6cc8ad6740f31728d55661181c8116e12c1dd" exitCode=60 Nov 03 22:45:21.498782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.498615 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" event={"ID":"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164","Type":"ContainerDied","Data":"d7a07a5a3e24739f9c8fd6542ba6cc8ad6740f31728d55661181c8116e12c1dd"} Nov 03 22:45:21.500626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.500603 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" event={"ID":"9445c683-e8c3-41bf-9cbc-5d3093d840c6","Type":"ContainerDied","Data":"6934d936ad0acd825bf6cdc6c998188244e34754c19ccafdc8df37df8455d376"} Nov 03 22:45:21.500928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.500910 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5" Nov 03 22:45:21.505344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.505320 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" event={"ID":"3ec653a6-42f1-49b3-a7d9-827945ef8946","Type":"ContainerStarted","Data":"2b47791b3210f979f256870f6e71f28eb23bd6d964f91ce5cb619f9623029da2"} Nov 03 22:45:21.509170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.509145 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" event={"ID":"2f271965-1d01-4f06-8619-ac80aaeb2154","Type":"ContainerDied","Data":"e843963bee2d77c83e8df5023361362c5daad0259e3b3efc18f3c5511f7f407d"} Nov 03 22:45:21.510460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.509355 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn" Nov 03 22:45:21.512694 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.511898 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" event={"ID":"b17306d5-2b3b-4d30-9a47-14920036a2ef","Type":"ContainerStarted","Data":"a0011e156bedaff4dd81705d7850c81ce7dd2060c1ffbf7a38084bf1c307ccb7"} Nov 03 22:45:21.516875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.516251 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" event={"ID":"c9d6c159-15e1-4fc7-8d50-40758bd126be","Type":"ContainerStarted","Data":"bf2446151a7a2ec820e49cee6cadcdaa6603b6e9adabb29fe085f81e66fa07e3"} Nov 03 22:45:21.520776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.520737 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" event={"ID":"c63f0f29-03cf-4880-8e34-b44f25a1eb2a","Type":"ContainerDied","Data":"03fc769d8cfb099f6c5c6cf70520caccc8867324be7d41402f6195449826e485"} Nov 03 22:45:21.521042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.520971 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p" Nov 03 22:45:21.534580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.534546 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d"] Nov 03 22:45:21.535438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.535414 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" event={"ID":"663b664e-98af-4583-aed0-da9572e83f52","Type":"ContainerStarted","Data":"7f1fe93f85915d0bd7e7aca564a4e8ef0527e2922b7185f6b6c4ee1031a37e59"} Nov 03 22:45:21.545667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.543499 4903 scope.go:122] "RemoveContainer" containerID="028c09f8ff7f38e8b0a631ea28f8ab00a8f8e29485b0409f8b8e5b46edce0d28" Nov 03 22:45:21.564532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.564507 4903 generic.go:356] "Generic (PLEG): container finished" podID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerID="33eeca89fa04e69b621016894383fa107a10f4d397f777366d0e49a0b7eeb790" exitCode=60 Nov 03 22:45:21.564674 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.564654 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" event={"ID":"b052a69f-e1a3-408e-9881-94770cb92ed6","Type":"ContainerDied","Data":"33eeca89fa04e69b621016894383fa107a10f4d397f777366d0e49a0b7eeb790"} Nov 03 22:45:21.565257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.565233 4903 scope.go:122] "RemoveContainer" containerID="33eeca89fa04e69b621016894383fa107a10f4d397f777366d0e49a0b7eeb790" Nov 03 22:45:21.565695 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:21.565671 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4_odh-ai-gateway-infra(b052a69f-e1a3-408e-9881-94770cb92ed6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" Nov 03 22:45:21.568536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.568510 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" event={"ID":"1010f884-c3b1-442b-bd6d-848666e3bfb4","Type":"ContainerStarted","Data":"805f7e3139fa48f2b4825b118e54e4c9c7b0dad14cb462b5996f365848cac4e4"} Nov 03 22:45:21.569101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.569081 4903 scope.go:122] "RemoveContainer" containerID="805f7e3139fa48f2b4825b118e54e4c9c7b0dad14cb462b5996f365848cac4e4" Nov 03 22:45:21.569770 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:21.569746 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv_odh-ai-gateway-infra(1010f884-c3b1-442b-bd6d-848666e3bfb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" Nov 03 22:45:21.596978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.596914 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" podStartSLOduration=2.596893585 podStartE2EDuration="2.596893585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:21.585484899 +0000 UTC m=+1947.989425878" watchObservedRunningTime="2026-11-03 22:45:21.596893585 +0000 UTC m=+1948.000834561" Nov 03 22:45:21.814310 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.814250 4903 scope.go:122] "RemoveContainer" containerID="361953a8e763ad902f87ffe83d45c439006f20dfd98de65e0638566135101f75" Nov 03 22:45:21.916348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.916315 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b686411-6854-4f2a-9400-8632d8e55009" path="/var/lib/kubelet/pods/1b686411-6854-4f2a-9400-8632d8e55009/volumes" Nov 03 22:45:21.917641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.917617 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" path="/var/lib/kubelet/pods/1d9725d5-d50c-496b-834e-a783e0c47b28/volumes" Nov 03 22:45:21.918411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.918392 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p"] Nov 03 22:45:21.918555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.918540 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895765-bpc7p"] Nov 03 22:45:21.996940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:21.996901 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:22.005926 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.005895 4903 scope.go:122] "RemoveContainer" containerID="aa131a0d93080197515072f519134ff833e3f49a3124b28d814cc5f7d46baa0f" Nov 03 22:45:22.016084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.016058 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:22.016259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.016071 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-kube-api-access-vrqsq\" (UniqueName: \"kubernetes.io/projected/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-kube-api-access-vrqsq\") pod \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\" (UID: \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\") " Nov 03 22:45:22.016385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.016354 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-maas-api-service-ca\") pod \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\" (UID: \"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164\") " Nov 03 22:45:22.016798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.016775 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-maas-api-service-ca" pod "3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" (UID: "3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:22.017324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.017287 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:22.018732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.018486 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-kube-api-access-vrqsq" pod "3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" (UID: "3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164"). InnerVolumeSpecName "kube-api-access-vrqsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:22.051369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.051336 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5"] Nov 03 22:45:22.063012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.062892 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb1fa5-29895765-8wrx5"] Nov 03 22:45:22.067709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.067681 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:22.068609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.068592 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:22.069608 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.069528 4903 scope.go:122] "RemoveContainer" containerID="86a8a3f6be964e4f93ec1c674accebadbf6860cfe792083efdbf4699cf7261a8" Nov 03 22:45:22.071375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.071354 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:45:22.072217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.072202 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:22.082072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.082054 4903 scope.go:122] "RemoveContainer" containerID="b5cd8682f0f58c75bf1d25530ab6f266dee97cb2283859bd0eba1eff81230dff" Nov 03 22:45:22.118632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.118610 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-kube-api-access-7lnj6\" (UniqueName: \"kubernetes.io/projected/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-kube-api-access-7lnj6\") pod \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\" (UID: \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\") " Nov 03 22:45:22.118748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.118674 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb962f11-d246-41e8-a596-6383a2d91a58-kube-api-access-thns8\" (UniqueName: \"kubernetes.io/projected/fb962f11-d246-41e8-a596-6383a2d91a58-kube-api-access-thns8\") pod \"fb962f11-d246-41e8-a596-6383a2d91a58\" (UID: \"fb962f11-d246-41e8-a596-6383a2d91a58\") " Nov 03 22:45:22.118748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.118726 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-maas-api-service-ca\") pod \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\" (UID: \"d177e30a-9ef2-4d4d-a2be-3872a6bc9411\") " Nov 03 22:45:22.118872 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.118857 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb962f11-d246-41e8-a596-6383a2d91a58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb962f11-d246-41e8-a596-6383a2d91a58-maas-api-service-ca\") pod \"fb962f11-d246-41e8-a596-6383a2d91a58\" (UID: \"fb962f11-d246-41e8-a596-6383a2d91a58\") " Nov 03 22:45:22.119163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.119136 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-maas-api-service-ca" pod "d177e30a-9ef2-4d4d-a2be-3872a6bc9411" (UID: "d177e30a-9ef2-4d4d-a2be-3872a6bc9411"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:22.119256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.119229 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb962f11-d246-41e8-a596-6383a2d91a58-maas-api-service-ca" pod "fb962f11-d246-41e8-a596-6383a2d91a58" (UID: "fb962f11-d246-41e8-a596-6383a2d91a58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:22.120013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.119944 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb962f11-d246-41e8-a596-6383a2d91a58-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:22.120013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.119974 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrqsq\" (UniqueName: \"kubernetes.io/projected/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164-kube-api-access-vrqsq\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:22.120013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.119990 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:22.120885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.120860 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb962f11-d246-41e8-a596-6383a2d91a58-kube-api-access-thns8" pod "fb962f11-d246-41e8-a596-6383a2d91a58" (UID: "fb962f11-d246-41e8-a596-6383a2d91a58"). InnerVolumeSpecName "kube-api-access-thns8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:22.121025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.121005 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-kube-api-access-7lnj6" pod "d177e30a-9ef2-4d4d-a2be-3872a6bc9411" (UID: "d177e30a-9ef2-4d4d-a2be-3872a6bc9411"). InnerVolumeSpecName "kube-api-access-7lnj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:22.123981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.123959 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn"] Nov 03 22:45:22.129608 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.129590 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fe7ca0-29895765-jt8qn"] Nov 03 22:45:22.222263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.222230 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lnj6\" (UniqueName: \"kubernetes.io/projected/d177e30a-9ef2-4d4d-a2be-3872a6bc9411-kube-api-access-7lnj6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:22.222263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.222265 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thns8\" (UniqueName: \"kubernetes.io/projected/fb962f11-d246-41e8-a596-6383a2d91a58-kube-api-access-thns8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:22.312092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.312040 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:22.585368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.585339 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" Nov 03 22:45:22.585368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.585350 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q" event={"ID":"3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164","Type":"ContainerDied","Data":"8049f64151c407c6fa9f090c4bf348fa6c5989e49fbd3a57f660c64355452ca0"} Nov 03 22:45:22.585551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.585418 4903 scope.go:122] "RemoveContainer" containerID="d7a07a5a3e24739f9c8fd6542ba6cc8ad6740f31728d55661181c8116e12c1dd" Nov 03 22:45:22.589413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.589384 4903 generic.go:356] "Generic (PLEG): container finished" podID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerID="2b47791b3210f979f256870f6e71f28eb23bd6d964f91ce5cb619f9623029da2" exitCode=60 Nov 03 22:45:22.589550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.589419 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" event={"ID":"3ec653a6-42f1-49b3-a7d9-827945ef8946","Type":"ContainerDied","Data":"2b47791b3210f979f256870f6e71f28eb23bd6d964f91ce5cb619f9623029da2"} Nov 03 22:45:22.589885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.589868 4903 scope.go:122] "RemoveContainer" containerID="2b47791b3210f979f256870f6e71f28eb23bd6d964f91ce5cb619f9623029da2" Nov 03 22:45:22.590270 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.590237 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps_odh-ai-gateway-infra(3ec653a6-42f1-49b3-a7d9-827945ef8946)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" Nov 03 22:45:22.592355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.592330 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" event={"ID":"b17306d5-2b3b-4d30-9a47-14920036a2ef","Type":"ContainerStarted","Data":"dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b"} Nov 03 22:45:22.594710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.594689 4903 generic.go:356] "Generic (PLEG): container finished" podID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerID="e986251d1224641fafd7eadebb47b9fa973a553ceb4066482af89b0642ab8e55" exitCode=60 Nov 03 22:45:22.594810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.594759 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" event={"ID":"31ad238a-db65-4160-a25b-9e29df9dbc94","Type":"ContainerDied","Data":"e986251d1224641fafd7eadebb47b9fa973a553ceb4066482af89b0642ab8e55"} Nov 03 22:45:22.595147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.595126 4903 scope.go:122] "RemoveContainer" containerID="e986251d1224641fafd7eadebb47b9fa973a553ceb4066482af89b0642ab8e55" Nov 03 22:45:22.595538 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.595513 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r_odh-ai-gateway-infra(31ad238a-db65-4160-a25b-9e29df9dbc94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" Nov 03 22:45:22.597176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.597151 4903 generic.go:356] "Generic (PLEG): container finished" podID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerID="c6fb4b64cd908d9a60876d32f0b286d34aa495f87b627d426ceccd87f20cb1e2" exitCode=60 Nov 03 22:45:22.597277 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.597206 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" event={"ID":"4e3fdb74-d353-4007-a96b-3af3db01cc57","Type":"ContainerDied","Data":"c6fb4b64cd908d9a60876d32f0b286d34aa495f87b627d426ceccd87f20cb1e2"} Nov 03 22:45:22.597743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.597577 4903 scope.go:122] "RemoveContainer" containerID="c6fb4b64cd908d9a60876d32f0b286d34aa495f87b627d426ceccd87f20cb1e2" Nov 03 22:45:22.598069 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.597899 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62_odh-ai-gateway-infra(4e3fdb74-d353-4007-a96b-3af3db01cc57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" Nov 03 22:45:22.600167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.600147 4903 generic.go:356] "Generic (PLEG): container finished" podID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerID="bf2446151a7a2ec820e49cee6cadcdaa6603b6e9adabb29fe085f81e66fa07e3" exitCode=60 Nov 03 22:45:22.600252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.600221 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" event={"ID":"c9d6c159-15e1-4fc7-8d50-40758bd126be","Type":"ContainerDied","Data":"bf2446151a7a2ec820e49cee6cadcdaa6603b6e9adabb29fe085f81e66fa07e3"} Nov 03 22:45:22.600637 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.600616 4903 scope.go:122] "RemoveContainer" containerID="bf2446151a7a2ec820e49cee6cadcdaa6603b6e9adabb29fe085f81e66fa07e3" Nov 03 22:45:22.600965 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.600943 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg_odh-ai-gateway-infra(c9d6c159-15e1-4fc7-8d50-40758bd126be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" Nov 03 22:45:22.602576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.602562 4903 scope.go:122] "RemoveContainer" containerID="9ef92eaa7acdd3a7b4ec388df510294a2db2296620866929e5d7f46296e22ee3" Nov 03 22:45:22.603810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.603789 4903 generic.go:356] "Generic (PLEG): container finished" podID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerID="080ac996d6169cc32f9ead2fe9b2846e0b5bd639dcb1a7c438f53b4ce944b468" exitCode=60 Nov 03 22:45:22.603905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.603862 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" event={"ID":"b10ff0e7-d241-428a-9e02-ac902f55b8c1","Type":"ContainerDied","Data":"080ac996d6169cc32f9ead2fe9b2846e0b5bd639dcb1a7c438f53b4ce944b468"} Nov 03 22:45:22.604243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.604224 4903 scope.go:122] "RemoveContainer" containerID="080ac996d6169cc32f9ead2fe9b2846e0b5bd639dcb1a7c438f53b4ce944b468" Nov 03 22:45:22.604573 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.604550 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr_odh-ai-gateway-infra(b10ff0e7-d241-428a-9e02-ac902f55b8c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" Nov 03 22:45:22.605892 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.605874 4903 generic.go:356] "Generic (PLEG): container finished" podID="663b664e-98af-4583-aed0-da9572e83f52" containerID="7f1fe93f85915d0bd7e7aca564a4e8ef0527e2922b7185f6b6c4ee1031a37e59" exitCode=60 Nov 03 22:45:22.605964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.605919 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" event={"ID":"663b664e-98af-4583-aed0-da9572e83f52","Type":"ContainerDied","Data":"7f1fe93f85915d0bd7e7aca564a4e8ef0527e2922b7185f6b6c4ee1031a37e59"} Nov 03 22:45:22.606254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.606236 4903 scope.go:122] "RemoveContainer" containerID="7f1fe93f85915d0bd7e7aca564a4e8ef0527e2922b7185f6b6c4ee1031a37e59" Nov 03 22:45:22.606570 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.606536 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj_odh-ai-gateway-infra(663b664e-98af-4583-aed0-da9572e83f52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" podUID="663b664e-98af-4583-aed0-da9572e83f52" Nov 03 22:45:22.608026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.608004 4903 generic.go:356] "Generic (PLEG): container finished" podID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerID="dda6484f03e0099ede659aa5ab2de2c9398e62614337987557b6685305dcbb98" exitCode=60 Nov 03 22:45:22.608026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.608026 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" event={"ID":"659a4a6e-2d77-4d9f-8284-a29be2d2de56","Type":"ContainerDied","Data":"dda6484f03e0099ede659aa5ab2de2c9398e62614337987557b6685305dcbb98"} Nov 03 22:45:22.608403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.608383 4903 scope.go:122] "RemoveContainer" containerID="dda6484f03e0099ede659aa5ab2de2c9398e62614337987557b6685305dcbb98" Nov 03 22:45:22.608678 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.608657 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s_odh-ai-gateway-infra(659a4a6e-2d77-4d9f-8284-a29be2d2de56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" Nov 03 22:45:22.609643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.609624 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" Nov 03 22:45:22.609742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.609647 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw" event={"ID":"d177e30a-9ef2-4d4d-a2be-3872a6bc9411","Type":"ContainerDied","Data":"0bbe19079a42761847aaeb47ee364f73206a958e39b0e372aa9ada06f863ab2d"} Nov 03 22:45:22.612325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.612306 4903 generic.go:356] "Generic (PLEG): container finished" podID="85615752-db5a-4dea-bc86-290e793928e0" containerID="03d8858ad533f5437eff43dea399176d70f2d07bace64ceea475254316104cdd" exitCode=60 Nov 03 22:45:22.612441 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.612343 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" event={"ID":"85615752-db5a-4dea-bc86-290e793928e0","Type":"ContainerDied","Data":"03d8858ad533f5437eff43dea399176d70f2d07bace64ceea475254316104cdd"} Nov 03 22:45:22.612796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.612779 4903 scope.go:122] "RemoveContainer" containerID="03d8858ad533f5437eff43dea399176d70f2d07bace64ceea475254316104cdd" Nov 03 22:45:22.613643 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.613618 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp_odh-ai-gateway-infra(85615752-db5a-4dea-bc86-290e793928e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" podUID="85615752-db5a-4dea-bc86-290e793928e0" Nov 03 22:45:22.616445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.616422 4903 generic.go:356] "Generic (PLEG): container finished" podID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerID="0894c4e5d9c94f82b38d0d27d37e941a09961c70bc8af31a5ae09b0bc7112d5b" exitCode=60 Nov 03 22:45:22.616555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.616496 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" event={"ID":"7c71cca5-adfa-4f08-9233-2719cadc9550","Type":"ContainerDied","Data":"0894c4e5d9c94f82b38d0d27d37e941a09961c70bc8af31a5ae09b0bc7112d5b"} Nov 03 22:45:22.616877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.616856 4903 scope.go:122] "RemoveContainer" containerID="0894c4e5d9c94f82b38d0d27d37e941a09961c70bc8af31a5ae09b0bc7112d5b" Nov 03 22:45:22.617199 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.617176 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh_odh-ai-gateway-infra(7c71cca5-adfa-4f08-9233-2719cadc9550)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" Nov 03 22:45:22.620420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.620394 4903 generic.go:356] "Generic (PLEG): container finished" podID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerID="805f7e3139fa48f2b4825b118e54e4c9c7b0dad14cb462b5996f365848cac4e4" exitCode=60 Nov 03 22:45:22.620667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.620614 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" event={"ID":"1010f884-c3b1-442b-bd6d-848666e3bfb4","Type":"ContainerDied","Data":"805f7e3139fa48f2b4825b118e54e4c9c7b0dad14cb462b5996f365848cac4e4"} Nov 03 22:45:22.621449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.620955 4903 scope.go:122] "RemoveContainer" containerID="805f7e3139fa48f2b4825b118e54e4c9c7b0dad14cb462b5996f365848cac4e4" Nov 03 22:45:22.621449 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:22.621226 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv_odh-ai-gateway-infra(1010f884-c3b1-442b-bd6d-848666e3bfb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" Nov 03 22:45:22.621449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.621378 4903 scope.go:122] "RemoveContainer" containerID="43358d3bc9debb1679337cf29be821b4a2bc01f496dd7d01317366a582ff7380" Nov 03 22:45:22.625987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.625863 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" Nov 03 22:45:22.625987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.625921 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p" event={"ID":"fb962f11-d246-41e8-a596-6383a2d91a58","Type":"ContainerDied","Data":"789cad193ebcd275424bb5cef0802f4110f12443829adafd232ff9c0ca39e1b6"} Nov 03 22:45:22.644754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.644733 4903 scope.go:122] "RemoveContainer" containerID="61f9aab06d7e39cc3ebabeab31aa3e47c05b6617a55c9e1fa04a450280e1a2fc" Nov 03 22:45:22.654737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.654711 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg"] Nov 03 22:45:22.662003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.661974 4903 scope.go:122] "RemoveContainer" containerID="5ac015103701def7a9eac439e282a47193b9adf15b9e6ed97edd79af5cffba83" Nov 03 22:45:22.682267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.681648 4903 scope.go:122] "RemoveContainer" containerID="4b98ad0cb672aba4321624a9249554ddc3d995c2f6eb689bc7c2e98f120d1aba" Nov 03 22:45:22.726145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.726033 4903 scope.go:122] "RemoveContainer" containerID="5427e22b12520acb7a80a3dc26c037e893eef49ea254b17ffc84618fa971c2e1" Nov 03 22:45:22.776151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.776064 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps"] Nov 03 22:45:22.780514 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.780491 4903 scope.go:122] "RemoveContainer" containerID="3040426ac54dd251b5e5ca081b54b45d23c3a2d6a107f4ae6e46f77621491ec3" Nov 03 22:45:22.795006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.794986 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:22.799346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.799088 4903 scope.go:122] "RemoveContainer" containerID="fe274ea4319ec218295251e145721df46e121c0a24a9d319190e26e1d7dd1b88" Nov 03 22:45:22.810266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.810246 4903 scope.go:122] "RemoveContainer" containerID="b88f4818986ec455b0d0a9cd309803b0f0ae97aef0607b5351a0fe60f77405bf" Nov 03 22:45:22.823676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.823651 4903 scope.go:122] "RemoveContainer" containerID="86434e39ef5458fe446d09f9df003b6091404d556697220e2df4c6b8992fe064" Nov 03 22:45:22.833170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.833147 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598378e1-2604-411f-9b3c-e2a196b4dc2e-kube-api-access-sxzz2\" (UniqueName: \"kubernetes.io/projected/598378e1-2604-411f-9b3c-e2a196b4dc2e-kube-api-access-sxzz2\") pod \"598378e1-2604-411f-9b3c-e2a196b4dc2e\" (UID: \"598378e1-2604-411f-9b3c-e2a196b4dc2e\") " Nov 03 22:45:22.833310 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.833274 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598378e1-2604-411f-9b3c-e2a196b4dc2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598378e1-2604-411f-9b3c-e2a196b4dc2e-maas-api-service-ca\") pod \"598378e1-2604-411f-9b3c-e2a196b4dc2e\" (UID: \"598378e1-2604-411f-9b3c-e2a196b4dc2e\") " Nov 03 22:45:22.833600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.833579 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598378e1-2604-411f-9b3c-e2a196b4dc2e-maas-api-service-ca" pod "598378e1-2604-411f-9b3c-e2a196b4dc2e" (UID: "598378e1-2604-411f-9b3c-e2a196b4dc2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:22.834174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.834152 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/598378e1-2604-411f-9b3c-e2a196b4dc2e-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:22.835263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.835243 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598378e1-2604-411f-9b3c-e2a196b4dc2e-kube-api-access-sxzz2" pod "598378e1-2604-411f-9b3c-e2a196b4dc2e" (UID: "598378e1-2604-411f-9b3c-e2a196b4dc2e"). InnerVolumeSpecName "kube-api-access-sxzz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:22.836789 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.836768 4903 scope.go:122] "RemoveContainer" containerID="b48831d65fc923bbcbb377a8026c1d6eac310556cbfcbfac414e94b495314c39" Nov 03 22:45:22.850815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.850790 4903 scope.go:122] "RemoveContainer" containerID="61287a576885fa52da1bf1ee3fa979ebe0be10405172183633d61553ba2abedf" Nov 03 22:45:22.915970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.915938 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q"] Nov 03 22:45:22.924913 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.924879 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4"] Nov 03 22:45:22.929827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.929800 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e19850-29895765-m6s2q"] Nov 03 22:45:22.936267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.936242 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxzz2\" (UniqueName: \"kubernetes.io/projected/598378e1-2604-411f-9b3c-e2a196b4dc2e-kube-api-access-sxzz2\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:22.968464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:22.968434 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv"] Nov 03 22:45:23.038121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.038008 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj"] Nov 03 22:45:23.141371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.135371 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p"] Nov 03 22:45:23.141371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.141045 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895765-mq44p"] Nov 03 22:45:23.154550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.154527 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:23.168188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.168162 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw"] Nov 03 22:45:23.173657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.173636 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-23d281-29895765-krhrw"] Nov 03 22:45:23.243906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.243814 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b052a69f-e1a3-408e-9881-94770cb92ed6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b052a69f-e1a3-408e-9881-94770cb92ed6-maas-api-service-ca\") pod \"b052a69f-e1a3-408e-9881-94770cb92ed6\" (UID: \"b052a69f-e1a3-408e-9881-94770cb92ed6\") " Nov 03 22:45:23.244183 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.244160 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b052a69f-e1a3-408e-9881-94770cb92ed6-kube-api-access-9lx7g\" (UniqueName: \"kubernetes.io/projected/b052a69f-e1a3-408e-9881-94770cb92ed6-kube-api-access-9lx7g\") pod \"b052a69f-e1a3-408e-9881-94770cb92ed6\" (UID: \"b052a69f-e1a3-408e-9881-94770cb92ed6\") " Nov 03 22:45:23.245207 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.245181 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b052a69f-e1a3-408e-9881-94770cb92ed6-maas-api-service-ca" pod "b052a69f-e1a3-408e-9881-94770cb92ed6" (UID: "b052a69f-e1a3-408e-9881-94770cb92ed6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:23.247394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.247312 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b052a69f-e1a3-408e-9881-94770cb92ed6-kube-api-access-9lx7g" pod "b052a69f-e1a3-408e-9881-94770cb92ed6" (UID: "b052a69f-e1a3-408e-9881-94770cb92ed6"). InnerVolumeSpecName "kube-api-access-9lx7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:23.346935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.346850 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b052a69f-e1a3-408e-9881-94770cb92ed6-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:23.346935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.346883 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lx7g\" (UniqueName: \"kubernetes.io/projected/b052a69f-e1a3-408e-9881-94770cb92ed6-kube-api-access-9lx7g\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:23.638596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.638511 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" Nov 03 22:45:23.638760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.638513 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d" event={"ID":"598378e1-2604-411f-9b3c-e2a196b4dc2e","Type":"ContainerDied","Data":"35bdda3520298e286ef4fb95ea082cba113be3a6bcb1b52a8e49477494a7a28b"} Nov 03 22:45:23.638760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.638654 4903 scope.go:122] "RemoveContainer" containerID="7aa8df1636df61a58b2b2cc344da2a19a07f41fc623c7f277e726f4abec36876" Nov 03 22:45:23.665124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.661077 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" Nov 03 22:45:23.665124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.661093 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4" event={"ID":"b052a69f-e1a3-408e-9881-94770cb92ed6","Type":"ContainerDied","Data":"7acc087cfc604a80fab3edfeb7773aca5a6256df384fdb379e0122df17b4a5e4"} Nov 03 22:45:23.665124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.661176 4903 scope.go:122] "RemoveContainer" containerID="33eeca89fa04e69b621016894383fa107a10f4d397f777366d0e49a0b7eeb790" Nov 03 22:45:23.676865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.676838 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp"] Nov 03 22:45:23.722858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.722828 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r"] Nov 03 22:45:23.831852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.829027 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" path="/var/lib/kubelet/pods/2f271965-1d01-4f06-8619-ac80aaeb2154/volumes" Nov 03 22:45:23.831852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.829685 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" path="/var/lib/kubelet/pods/3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164/volumes" Nov 03 22:45:23.831852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.830332 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" path="/var/lib/kubelet/pods/9445c683-e8c3-41bf-9cbc-5d3093d840c6/volumes" Nov 03 22:45:23.831852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.831491 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" path="/var/lib/kubelet/pods/c63f0f29-03cf-4880-8e34-b44f25a1eb2a/volumes" Nov 03 22:45:23.832191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.832133 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" path="/var/lib/kubelet/pods/d177e30a-9ef2-4d4d-a2be-3872a6bc9411/volumes" Nov 03 22:45:23.832945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.832874 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" path="/var/lib/kubelet/pods/fb962f11-d246-41e8-a596-6383a2d91a58/volumes" Nov 03 22:45:23.833855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.833829 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d"] Nov 03 22:45:23.835833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.835659 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-50d738-29895765-fxz8d"] Nov 03 22:45:23.881549 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.862861 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr"] Nov 03 22:45:23.893986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:23.893905 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62"] Nov 03 22:45:24.006543 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.006499 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4"] Nov 03 22:45:24.016982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.016946 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:24.017120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.017106 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895765-pppz4"] Nov 03 22:45:24.019719 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.019690 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:24.032218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.031968 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s"] Nov 03 22:45:24.040314 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:24.040267 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:24.063206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.062911 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh"] Nov 03 22:45:24.081485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.081132 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec653a6-42f1-49b3-a7d9-827945ef8946-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec653a6-42f1-49b3-a7d9-827945ef8946-maas-api-service-ca\") pod \"3ec653a6-42f1-49b3-a7d9-827945ef8946\" (UID: \"3ec653a6-42f1-49b3-a7d9-827945ef8946\") " Nov 03 22:45:24.081485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.081278 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d6c159-15e1-4fc7-8d50-40758bd126be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6c159-15e1-4fc7-8d50-40758bd126be-maas-api-service-ca\") pod \"c9d6c159-15e1-4fc7-8d50-40758bd126be\" (UID: \"c9d6c159-15e1-4fc7-8d50-40758bd126be\") " Nov 03 22:45:24.081485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.081418 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec653a6-42f1-49b3-a7d9-827945ef8946-kube-api-access-fvfv5\" (UniqueName: \"kubernetes.io/projected/3ec653a6-42f1-49b3-a7d9-827945ef8946-kube-api-access-fvfv5\") pod \"3ec653a6-42f1-49b3-a7d9-827945ef8946\" (UID: \"3ec653a6-42f1-49b3-a7d9-827945ef8946\") " Nov 03 22:45:24.081684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.081652 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d6c159-15e1-4fc7-8d50-40758bd126be-kube-api-access-whh9n\" (UniqueName: \"kubernetes.io/projected/c9d6c159-15e1-4fc7-8d50-40758bd126be-kube-api-access-whh9n\") pod \"c9d6c159-15e1-4fc7-8d50-40758bd126be\" (UID: \"c9d6c159-15e1-4fc7-8d50-40758bd126be\") " Nov 03 22:45:24.090676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.082238 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d6c159-15e1-4fc7-8d50-40758bd126be-maas-api-service-ca" pod "c9d6c159-15e1-4fc7-8d50-40758bd126be" (UID: "c9d6c159-15e1-4fc7-8d50-40758bd126be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.090676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.082572 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d6c159-15e1-4fc7-8d50-40758bd126be-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.090676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.083037 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec653a6-42f1-49b3-a7d9-827945ef8946-maas-api-service-ca" pod "3ec653a6-42f1-49b3-a7d9-827945ef8946" (UID: "3ec653a6-42f1-49b3-a7d9-827945ef8946"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.106321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.106231 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec653a6-42f1-49b3-a7d9-827945ef8946-kube-api-access-fvfv5" pod "3ec653a6-42f1-49b3-a7d9-827945ef8946" (UID: "3ec653a6-42f1-49b3-a7d9-827945ef8946"). InnerVolumeSpecName "kube-api-access-fvfv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.109015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.108978 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d6c159-15e1-4fc7-8d50-40758bd126be-kube-api-access-whh9n" pod "c9d6c159-15e1-4fc7-8d50-40758bd126be" (UID: "c9d6c159-15e1-4fc7-8d50-40758bd126be"). InnerVolumeSpecName "kube-api-access-whh9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.196411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.196272 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec653a6-42f1-49b3-a7d9-827945ef8946-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.196411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.196322 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvfv5\" (UniqueName: \"kubernetes.io/projected/3ec653a6-42f1-49b3-a7d9-827945ef8946-kube-api-access-fvfv5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.196411 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.196337 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whh9n\" (UniqueName: \"kubernetes.io/projected/c9d6c159-15e1-4fc7-8d50-40758bd126be-kube-api-access-whh9n\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.209456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.209429 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:24.249338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.236522 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:24.249338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.239725 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:24.250698 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.250676 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:24.312924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.312889 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663b664e-98af-4583-aed0-da9572e83f52-kube-api-access-g7sw5\" (UniqueName: \"kubernetes.io/projected/663b664e-98af-4583-aed0-da9572e83f52-kube-api-access-g7sw5\") pod \"663b664e-98af-4583-aed0-da9572e83f52\" (UID: \"663b664e-98af-4583-aed0-da9572e83f52\") " Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.323610 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663b664e-98af-4583-aed0-da9572e83f52-kube-api-access-g7sw5" pod "663b664e-98af-4583-aed0-da9572e83f52" (UID: "663b664e-98af-4583-aed0-da9572e83f52"). InnerVolumeSpecName "kube-api-access-g7sw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.330422 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85615752-db5a-4dea-bc86-290e793928e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85615752-db5a-4dea-bc86-290e793928e0-maas-api-service-ca\") pod \"85615752-db5a-4dea-bc86-290e793928e0\" (UID: \"85615752-db5a-4dea-bc86-290e793928e0\") " Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.330495 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ad238a-db65-4160-a25b-9e29df9dbc94-kube-api-access-6rf6f\" (UniqueName: \"kubernetes.io/projected/31ad238a-db65-4160-a25b-9e29df9dbc94-kube-api-access-6rf6f\") pod \"31ad238a-db65-4160-a25b-9e29df9dbc94\" (UID: \"31ad238a-db65-4160-a25b-9e29df9dbc94\") " Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.330583 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1010f884-c3b1-442b-bd6d-848666e3bfb4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f884-c3b1-442b-bd6d-848666e3bfb4-maas-api-service-ca\") pod \"1010f884-c3b1-442b-bd6d-848666e3bfb4\" (UID: \"1010f884-c3b1-442b-bd6d-848666e3bfb4\") " Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.330775 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/663b664e-98af-4583-aed0-da9572e83f52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b664e-98af-4583-aed0-da9572e83f52-maas-api-service-ca\") pod \"663b664e-98af-4583-aed0-da9572e83f52\" (UID: \"663b664e-98af-4583-aed0-da9572e83f52\") " Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.330827 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85615752-db5a-4dea-bc86-290e793928e0-kube-api-access-jgsf5\" (UniqueName: \"kubernetes.io/projected/85615752-db5a-4dea-bc86-290e793928e0-kube-api-access-jgsf5\") pod \"85615752-db5a-4dea-bc86-290e793928e0\" (UID: \"85615752-db5a-4dea-bc86-290e793928e0\") " Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.330781 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85615752-db5a-4dea-bc86-290e793928e0-maas-api-service-ca" pod "85615752-db5a-4dea-bc86-290e793928e0" (UID: "85615752-db5a-4dea-bc86-290e793928e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.330912 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ad238a-db65-4160-a25b-9e29df9dbc94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ad238a-db65-4160-a25b-9e29df9dbc94-maas-api-service-ca\") pod \"31ad238a-db65-4160-a25b-9e29df9dbc94\" (UID: \"31ad238a-db65-4160-a25b-9e29df9dbc94\") " Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.330952 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1010f884-c3b1-442b-bd6d-848666e3bfb4-maas-api-service-ca" pod "1010f884-c3b1-442b-bd6d-848666e3bfb4" (UID: "1010f884-c3b1-442b-bd6d-848666e3bfb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.330972 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1010f884-c3b1-442b-bd6d-848666e3bfb4-kube-api-access-gqpqv\" (UniqueName: \"kubernetes.io/projected/1010f884-c3b1-442b-bd6d-848666e3bfb4-kube-api-access-gqpqv\") pod \"1010f884-c3b1-442b-bd6d-848666e3bfb4\" (UID: \"1010f884-c3b1-442b-bd6d-848666e3bfb4\") " Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.331232 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663b664e-98af-4583-aed0-da9572e83f52-maas-api-service-ca" pod "663b664e-98af-4583-aed0-da9572e83f52" (UID: "663b664e-98af-4583-aed0-da9572e83f52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.331851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.331286 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ad238a-db65-4160-a25b-9e29df9dbc94-maas-api-service-ca" pod "31ad238a-db65-4160-a25b-9e29df9dbc94" (UID: "31ad238a-db65-4160-a25b-9e29df9dbc94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.332617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.331905 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7sw5\" (UniqueName: \"kubernetes.io/projected/663b664e-98af-4583-aed0-da9572e83f52-kube-api-access-g7sw5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.332617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.331931 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85615752-db5a-4dea-bc86-290e793928e0-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.332617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.331947 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1010f884-c3b1-442b-bd6d-848666e3bfb4-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.332617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.331961 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/663b664e-98af-4583-aed0-da9572e83f52-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.332617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.331975 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31ad238a-db65-4160-a25b-9e29df9dbc94-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.333846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.333807 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ad238a-db65-4160-a25b-9e29df9dbc94-kube-api-access-6rf6f" pod "31ad238a-db65-4160-a25b-9e29df9dbc94" (UID: "31ad238a-db65-4160-a25b-9e29df9dbc94"). InnerVolumeSpecName "kube-api-access-6rf6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.334641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.334618 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85615752-db5a-4dea-bc86-290e793928e0-kube-api-access-jgsf5" pod "85615752-db5a-4dea-bc86-290e793928e0" (UID: "85615752-db5a-4dea-bc86-290e793928e0"). InnerVolumeSpecName "kube-api-access-jgsf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.334719 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.334656 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1010f884-c3b1-442b-bd6d-848666e3bfb4-kube-api-access-gqpqv" pod "1010f884-c3b1-442b-bd6d-848666e3bfb4" (UID: "1010f884-c3b1-442b-bd6d-848666e3bfb4"). InnerVolumeSpecName "kube-api-access-gqpqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.339206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.339187 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:24.345070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.345049 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.445898 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3fdb74-d353-4007-a96b-3af3db01cc57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3fdb74-d353-4007-a96b-3af3db01cc57-maas-api-service-ca\") pod \"4e3fdb74-d353-4007-a96b-3af3db01cc57\" (UID: \"4e3fdb74-d353-4007-a96b-3af3db01cc57\") " Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.446068 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10ff0e7-d241-428a-9e02-ac902f55b8c1-kube-api-access-s5mhc\" (UniqueName: \"kubernetes.io/projected/b10ff0e7-d241-428a-9e02-ac902f55b8c1-kube-api-access-s5mhc\") pod \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\" (UID: \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\") " Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.446126 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3fdb74-d353-4007-a96b-3af3db01cc57-kube-api-access-2frhz\" (UniqueName: \"kubernetes.io/projected/4e3fdb74-d353-4007-a96b-3af3db01cc57-kube-api-access-2frhz\") pod \"4e3fdb74-d353-4007-a96b-3af3db01cc57\" (UID: \"4e3fdb74-d353-4007-a96b-3af3db01cc57\") " Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.446277 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3fdb74-d353-4007-a96b-3af3db01cc57-maas-api-service-ca" pod "4e3fdb74-d353-4007-a96b-3af3db01cc57" (UID: "4e3fdb74-d353-4007-a96b-3af3db01cc57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.446360 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10ff0e7-d241-428a-9e02-ac902f55b8c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ff0e7-d241-428a-9e02-ac902f55b8c1-maas-api-service-ca\") pod \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\" (UID: \"b10ff0e7-d241-428a-9e02-ac902f55b8c1\") " Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.447110 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10ff0e7-d241-428a-9e02-ac902f55b8c1-maas-api-service-ca" pod "b10ff0e7-d241-428a-9e02-ac902f55b8c1" (UID: "b10ff0e7-d241-428a-9e02-ac902f55b8c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.447346 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rf6f\" (UniqueName: \"kubernetes.io/projected/31ad238a-db65-4160-a25b-9e29df9dbc94-kube-api-access-6rf6f\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.447367 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ff0e7-d241-428a-9e02-ac902f55b8c1-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.447392 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgsf5\" (UniqueName: \"kubernetes.io/projected/85615752-db5a-4dea-bc86-290e793928e0-kube-api-access-jgsf5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.447406 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqpqv\" (UniqueName: \"kubernetes.io/projected/1010f884-c3b1-442b-bd6d-848666e3bfb4-kube-api-access-gqpqv\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.448279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.447420 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3fdb74-d353-4007-a96b-3af3db01cc57-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.451441 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.449908 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10ff0e7-d241-428a-9e02-ac902f55b8c1-kube-api-access-s5mhc" pod "b10ff0e7-d241-428a-9e02-ac902f55b8c1" (UID: "b10ff0e7-d241-428a-9e02-ac902f55b8c1"). InnerVolumeSpecName "kube-api-access-s5mhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.451441 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.450600 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3fdb74-d353-4007-a96b-3af3db01cc57-kube-api-access-2frhz" pod "4e3fdb74-d353-4007-a96b-3af3db01cc57" (UID: "4e3fdb74-d353-4007-a96b-3af3db01cc57"). InnerVolumeSpecName "kube-api-access-2frhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.457014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.456993 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:24.461782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.461762 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:24.548394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.548273 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c71cca5-adfa-4f08-9233-2719cadc9550-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c71cca5-adfa-4f08-9233-2719cadc9550-maas-api-service-ca\") pod \"7c71cca5-adfa-4f08-9233-2719cadc9550\" (UID: \"7c71cca5-adfa-4f08-9233-2719cadc9550\") " Nov 03 22:45:24.548394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.548350 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c71cca5-adfa-4f08-9233-2719cadc9550-kube-api-access-nwnpb\" (UniqueName: \"kubernetes.io/projected/7c71cca5-adfa-4f08-9233-2719cadc9550-kube-api-access-nwnpb\") pod \"7c71cca5-adfa-4f08-9233-2719cadc9550\" (UID: \"7c71cca5-adfa-4f08-9233-2719cadc9550\") " Nov 03 22:45:24.548663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.548516 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659a4a6e-2d77-4d9f-8284-a29be2d2de56-kube-api-access-rwrhp\" (UniqueName: \"kubernetes.io/projected/659a4a6e-2d77-4d9f-8284-a29be2d2de56-kube-api-access-rwrhp\") pod \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\" (UID: \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\") " Nov 03 22:45:24.548779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.548758 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659a4a6e-2d77-4d9f-8284-a29be2d2de56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659a4a6e-2d77-4d9f-8284-a29be2d2de56-maas-api-service-ca\") pod \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\" (UID: \"659a4a6e-2d77-4d9f-8284-a29be2d2de56\") " Nov 03 22:45:24.550187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.549552 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c71cca5-adfa-4f08-9233-2719cadc9550-maas-api-service-ca" pod "7c71cca5-adfa-4f08-9233-2719cadc9550" (UID: "7c71cca5-adfa-4f08-9233-2719cadc9550"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.550187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.549889 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659a4a6e-2d77-4d9f-8284-a29be2d2de56-maas-api-service-ca" pod "659a4a6e-2d77-4d9f-8284-a29be2d2de56" (UID: "659a4a6e-2d77-4d9f-8284-a29be2d2de56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:24.550187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.550105 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659a4a6e-2d77-4d9f-8284-a29be2d2de56-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.550187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.550127 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5mhc\" (UniqueName: \"kubernetes.io/projected/b10ff0e7-d241-428a-9e02-ac902f55b8c1-kube-api-access-s5mhc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.550187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.550142 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2frhz\" (UniqueName: \"kubernetes.io/projected/4e3fdb74-d353-4007-a96b-3af3db01cc57-kube-api-access-2frhz\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.550187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.550156 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c71cca5-adfa-4f08-9233-2719cadc9550-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.553745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.553702 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659a4a6e-2d77-4d9f-8284-a29be2d2de56-kube-api-access-rwrhp" pod "659a4a6e-2d77-4d9f-8284-a29be2d2de56" (UID: "659a4a6e-2d77-4d9f-8284-a29be2d2de56"). InnerVolumeSpecName "kube-api-access-rwrhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.554436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.554414 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c71cca5-adfa-4f08-9233-2719cadc9550-kube-api-access-nwnpb" pod "7c71cca5-adfa-4f08-9233-2719cadc9550" (UID: "7c71cca5-adfa-4f08-9233-2719cadc9550"). InnerVolumeSpecName "kube-api-access-nwnpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:24.652071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.652025 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrhp\" (UniqueName: \"kubernetes.io/projected/659a4a6e-2d77-4d9f-8284-a29be2d2de56-kube-api-access-rwrhp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.652071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.652073 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwnpb\" (UniqueName: \"kubernetes.io/projected/7c71cca5-adfa-4f08-9233-2719cadc9550-kube-api-access-nwnpb\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:24.690009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.689977 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" Nov 03 22:45:24.690555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.690384 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps" event={"ID":"3ec653a6-42f1-49b3-a7d9-827945ef8946","Type":"ContainerDied","Data":"6a7c73a93ede72590d76aecccbdbb22f3c9c51794305868605348071f6fce804"} Nov 03 22:45:24.690555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.690458 4903 scope.go:122] "RemoveContainer" containerID="2b47791b3210f979f256870f6e71f28eb23bd6d964f91ce5cb619f9623029da2" Nov 03 22:45:24.693437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.693404 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" event={"ID":"31ad238a-db65-4160-a25b-9e29df9dbc94","Type":"ContainerDied","Data":"17d5d8ff298769bf97b8615028d3ff5c3f8cf669769fb060cd40fe1d2940906d"} Nov 03 22:45:24.693571 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.693507 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r" Nov 03 22:45:24.700345 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.698811 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" Nov 03 22:45:24.700345 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.698827 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62" event={"ID":"4e3fdb74-d353-4007-a96b-3af3db01cc57","Type":"ContainerDied","Data":"e85b14f3519d78928d72225c1617f1c4a4487ffccd41aa41f7e9a58257d090bf"} Nov 03 22:45:24.707508 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.707484 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" Nov 03 22:45:24.707613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.707518 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg" event={"ID":"c9d6c159-15e1-4fc7-8d50-40758bd126be","Type":"ContainerDied","Data":"1f22291cc12bf648fc1009594b1bd7dff127901f80b1b5ddbf5532faa7c84f92"} Nov 03 22:45:24.707613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.707566 4903 scope.go:122] "RemoveContainer" containerID="e986251d1224641fafd7eadebb47b9fa973a553ceb4066482af89b0642ab8e55" Nov 03 22:45:24.710819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.710792 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" event={"ID":"b10ff0e7-d241-428a-9e02-ac902f55b8c1","Type":"ContainerDied","Data":"3a5aa15fc7cc861a75400318353c39a900121e2812cb8948c1cacb0e31399de1"} Nov 03 22:45:24.710954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.710883 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr" Nov 03 22:45:24.714753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.714730 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" event={"ID":"663b664e-98af-4583-aed0-da9572e83f52","Type":"ContainerDied","Data":"b6c2d65d8f9e41ad45aa312ee64941d6c47d7cecf892ac33dd4e8d58802ed84e"} Nov 03 22:45:24.714839 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.714822 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj" Nov 03 22:45:24.719930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.719908 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" event={"ID":"659a4a6e-2d77-4d9f-8284-a29be2d2de56","Type":"ContainerDied","Data":"358a1eecd4878a65d0e1a025a4fd691885b8e4765e4fd38a80452fb758e89d2c"} Nov 03 22:45:24.720037 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.719999 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s" Nov 03 22:45:24.722012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.721923 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" event={"ID":"85615752-db5a-4dea-bc86-290e793928e0","Type":"ContainerDied","Data":"48c07598dadf2f28ebe4ee2875e0734db98c64fffff4e89ca0994033a92d8454"} Nov 03 22:45:24.722114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.722012 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp" Nov 03 22:45:24.729771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.727645 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" Nov 03 22:45:24.729771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.727692 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh" event={"ID":"7c71cca5-adfa-4f08-9233-2719cadc9550","Type":"ContainerDied","Data":"fbddc9beb04d09ce0fa260b7b26cae8f36f60e2c71a4d6e979c4c1edae6a08f6"} Nov 03 22:45:24.732123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.732105 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" Nov 03 22:45:24.732229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.732132 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv" event={"ID":"1010f884-c3b1-442b-bd6d-848666e3bfb4","Type":"ContainerDied","Data":"ac05d9f0706a8766fd8b751f4540e25661289c7e9917cc2b1ec079824afa0c55"} Nov 03 22:45:24.743918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.743893 4903 scope.go:122] "RemoveContainer" containerID="c6fb4b64cd908d9a60876d32f0b286d34aa495f87b627d426ceccd87f20cb1e2" Nov 03 22:45:24.758744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.758722 4903 scope.go:122] "RemoveContainer" containerID="bf2446151a7a2ec820e49cee6cadcdaa6603b6e9adabb29fe085f81e66fa07e3" Nov 03 22:45:24.769544 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.769520 4903 scope.go:122] "RemoveContainer" containerID="080ac996d6169cc32f9ead2fe9b2846e0b5bd639dcb1a7c438f53b4ce944b468" Nov 03 22:45:24.783377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.783342 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r"] Nov 03 22:45:24.787436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.787415 4903 scope.go:122] "RemoveContainer" containerID="7f1fe93f85915d0bd7e7aca564a4e8ef0527e2922b7185f6b6c4ee1031a37e59" Nov 03 22:45:24.799857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.799833 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5187e1-29895765-r4k4r"] Nov 03 22:45:24.801825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.801804 4903 scope.go:122] "RemoveContainer" containerID="dda6484f03e0099ede659aa5ab2de2c9398e62614337987557b6685305dcbb98" Nov 03 22:45:24.811377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.811353 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg"] Nov 03 22:45:24.816670 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.816652 4903 scope.go:122] "RemoveContainer" containerID="03d8858ad533f5437eff43dea399176d70f2d07bace64ceea475254316104cdd" Nov 03 22:45:24.817524 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.817501 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ccdb60-29895765-dr2wg"] Nov 03 22:45:24.833627 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.833605 4903 scope.go:122] "RemoveContainer" containerID="0894c4e5d9c94f82b38d0d27d37e941a09961c70bc8af31a5ae09b0bc7112d5b" Nov 03 22:45:24.870010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.869976 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr"] Nov 03 22:45:24.872124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.872079 4903 scope.go:122] "RemoveContainer" containerID="805f7e3139fa48f2b4825b118e54e4c9c7b0dad14cb462b5996f365848cac4e4" Nov 03 22:45:24.877451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.877428 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895765-ftjlr"] Nov 03 22:45:24.908788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.908760 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62"] Nov 03 22:45:24.916237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.916208 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895765-zcx62"] Nov 03 22:45:24.948091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.948023 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps"] Nov 03 22:45:24.956164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:24.956129 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-71a7df-29895765-2spps"] Nov 03 22:45:25.012098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.012070 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj"] Nov 03 22:45:25.017072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.017044 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895765-k9mtj"] Nov 03 22:45:25.079252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.079221 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv"] Nov 03 22:45:25.088114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.088080 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-50b9b5-29895765-znppv"] Nov 03 22:45:25.120442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.120381 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s"] Nov 03 22:45:25.126404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.126375 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4b5e2b-29895765-5cn4s"] Nov 03 22:45:25.150481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.150455 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh"] Nov 03 22:45:25.155021 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.154997 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-341c51-29895765-rjxlh"] Nov 03 22:45:25.183852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.183792 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp"] Nov 03 22:45:25.188421 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.188399 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9e540d-29895765-tcnrp"] Nov 03 22:45:25.741577 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.741540 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" path="/var/lib/kubelet/pods/1010f884-c3b1-442b-bd6d-848666e3bfb4/volumes" Nov 03 22:45:25.742283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.742246 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" path="/var/lib/kubelet/pods/31ad238a-db65-4160-a25b-9e29df9dbc94/volumes" Nov 03 22:45:25.742939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.742915 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" path="/var/lib/kubelet/pods/3ec653a6-42f1-49b3-a7d9-827945ef8946/volumes" Nov 03 22:45:25.743991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.743962 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" path="/var/lib/kubelet/pods/4e3fdb74-d353-4007-a96b-3af3db01cc57/volumes" Nov 03 22:45:25.744646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.744596 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" path="/var/lib/kubelet/pods/598378e1-2604-411f-9b3c-e2a196b4dc2e/volumes" Nov 03 22:45:25.745711 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.745538 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" path="/var/lib/kubelet/pods/659a4a6e-2d77-4d9f-8284-a29be2d2de56/volumes" Nov 03 22:45:25.746639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.746617 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663b664e-98af-4583-aed0-da9572e83f52" path="/var/lib/kubelet/pods/663b664e-98af-4583-aed0-da9572e83f52/volumes" Nov 03 22:45:25.747259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.747241 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" path="/var/lib/kubelet/pods/7c71cca5-adfa-4f08-9233-2719cadc9550/volumes" Nov 03 22:45:25.747897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.747879 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85615752-db5a-4dea-bc86-290e793928e0" path="/var/lib/kubelet/pods/85615752-db5a-4dea-bc86-290e793928e0/volumes" Nov 03 22:45:25.749002 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.748981 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" path="/var/lib/kubelet/pods/b052a69f-e1a3-408e-9881-94770cb92ed6/volumes" Nov 03 22:45:25.749859 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.749599 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" path="/var/lib/kubelet/pods/b10ff0e7-d241-428a-9e02-ac902f55b8c1/volumes" Nov 03 22:45:25.751131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:25.750184 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" path="/var/lib/kubelet/pods/c9d6c159-15e1-4fc7-8d50-40758bd126be/volumes" Nov 03 22:45:28.324973 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:28.324931 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:29.652810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.652775 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7"] Nov 03 22:45:29.653731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653710 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerName="cleanup" Nov 03 22:45:29.653731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653729 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653742 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653749 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653764 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653773 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653781 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653787 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653798 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653802 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653812 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653817 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653823 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653827 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653836 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653840 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653849 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653853 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653863 4903 cpu_manager.go:414] "containerMap: removing container" podUID="663b664e-98af-4583-aed0-da9572e83f52" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653867 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663b664e-98af-4583-aed0-da9572e83f52" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653877 4903 cpu_manager.go:414] "containerMap: removing container" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653883 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653892 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653896 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653901 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653907 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerName="cleanup" Nov 03 22:45:29.653924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653939 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653944 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653950 4903 cpu_manager.go:414] "containerMap: removing container" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653956 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653966 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653973 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653984 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.653988 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654006 4903 cpu_manager.go:414] "containerMap: removing container" podUID="85615752-db5a-4dea-bc86-290e793928e0" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654011 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85615752-db5a-4dea-bc86-290e793928e0" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654017 4903 cpu_manager.go:414] "containerMap: removing container" podUID="663b664e-98af-4583-aed0-da9572e83f52" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654021 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663b664e-98af-4583-aed0-da9572e83f52" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654028 4903 cpu_manager.go:414] "containerMap: removing container" podUID="85615752-db5a-4dea-bc86-290e793928e0" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654033 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85615752-db5a-4dea-bc86-290e793928e0" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654044 4903 cpu_manager.go:414] "containerMap: removing container" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654050 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654058 4903 cpu_manager.go:414] "containerMap: removing container" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654062 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654067 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654071 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654077 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654081 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654091 4903 cpu_manager.go:414] "containerMap: removing container" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654096 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654101 4903 cpu_manager.go:414] "containerMap: removing container" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654105 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654113 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654118 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654124 4903 cpu_manager.go:414] "containerMap: removing container" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654129 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654134 4903 cpu_manager.go:414] "containerMap: removing container" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654138 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654146 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654151 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654160 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654164 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654171 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654175 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654189 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654193 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654202 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654207 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654217 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654221 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654228 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654232 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654239 4903 cpu_manager.go:414] "containerMap: removing container" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654242 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654251 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654256 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654269 4903 cpu_manager.go:414] "containerMap: removing container" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654276 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654283 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654288 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654311 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654316 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654325 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654330 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654339 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654344 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654352 4903 cpu_manager.go:414] "containerMap: removing container" podUID="663b664e-98af-4583-aed0-da9572e83f52" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654356 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663b664e-98af-4583-aed0-da9572e83f52" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654366 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654371 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654383 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654390 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654399 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654403 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654571 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654579 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654585 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654593 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654598 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654604 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654614 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654621 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654626 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="85615752-db5a-4dea-bc86-290e793928e0" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654632 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="663b664e-98af-4583-aed0-da9572e83f52" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654636 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="85615752-db5a-4dea-bc86-290e793928e0" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654641 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654647 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b686411-6854-4f2a-9400-8632d8e55009" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654652 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654658 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654662 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654667 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654673 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654679 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654685 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654693 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654702 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="598378e1-2604-411f-9b3c-e2a196b4dc2e" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654707 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654712 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654719 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654726 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654732 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654737 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654744 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec653a6-42f1-49b3-a7d9-827945ef8946" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654749 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654756 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654766 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654776 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="85615752-db5a-4dea-bc86-290e793928e0" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654782 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654787 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654794 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f271965-1d01-4f06-8619-ac80aaeb2154" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654799 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerName="cleanup" Nov 03 22:45:29.654753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654805 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654811 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654816 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="663b664e-98af-4583-aed0-da9572e83f52" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654822 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10ff0e7-d241-428a-9e02-ac902f55b8c1" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654829 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654835 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3fdb74-d353-4007-a96b-3af3db01cc57" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654841 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654853 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1010f884-c3b1-442b-bd6d-848666e3bfb4" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654859 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654865 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654870 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654877 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerName="cleanup" Nov 03 22:45:29.657939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.654882 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerName="cleanup" Nov 03 22:45:29.660078 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.660060 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.675575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.675550 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7"] Nov 03 22:45:29.679367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.679342 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:45:29.679484 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.679454 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:45:29.679561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.679485 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:45:29.679761 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.679734 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:45:29.748692 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.748653 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.748887 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.748776 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqzln\" (UniqueName: \"kubernetes.io/projected/8a83b01c-bd97-4f4d-bb55-1db21191f756-kube-api-access-mqzln\") pod \"maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.748954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.748937 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.860319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.854637 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.860319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.854740 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqzln\" (UniqueName: \"kubernetes.io/projected/8a83b01c-bd97-4f4d-bb55-1db21191f756-kube-api-access-mqzln\") pod \"maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.860319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.854880 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.860642 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.860416 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.864431 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.864393 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.882363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.882329 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqzln\" (UniqueName: \"kubernetes.io/projected/8a83b01c-bd97-4f4d-bb55-1db21191f756-kube-api-access-mqzln\") pod \"maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:29.975848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:29.975770 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:30.291791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:30.291762 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7"] Nov 03 22:45:30.311831 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:30.311805 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:30.325244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:30.325222 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:30.818138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:30.818098 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" event={"ID":"8a83b01c-bd97-4f4d-bb55-1db21191f756","Type":"ContainerStarted","Data":"c6d78eb1deb5249ac5bb14c5bd582bbce4d56aaf50357c9551597ddbc93e7d56"} Nov 03 22:45:30.818662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:30.818149 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" event={"ID":"8a83b01c-bd97-4f4d-bb55-1db21191f756","Type":"ContainerStarted","Data":"862af109432224536ea443e875c47e4cdecd8616dfad169bec5116ddf9aa2584"} Nov 03 22:45:30.829425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:30.829401 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:45:30.923561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:30.923516 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" podStartSLOduration=1.923501233 podStartE2EDuration="1.923501233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:30.88145558 +0000 UTC m=+1957.285396555" watchObservedRunningTime="2026-11-03 22:45:30.923501233 +0000 UTC m=+1957.327442207" Nov 03 22:45:31.829184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:31.829145 4903 generic.go:356] "Generic (PLEG): container finished" podID="8a83b01c-bd97-4f4d-bb55-1db21191f756" containerID="c6d78eb1deb5249ac5bb14c5bd582bbce4d56aaf50357c9551597ddbc93e7d56" exitCode=0 Nov 03 22:45:31.829590 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:31.829233 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" event={"ID":"8a83b01c-bd97-4f4d-bb55-1db21191f756","Type":"ContainerDied","Data":"c6d78eb1deb5249ac5bb14c5bd582bbce4d56aaf50357c9551597ddbc93e7d56"} Nov 03 22:45:32.456073 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:32.456029 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5"] Nov 03 22:45:32.456262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:32.456240 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" containerID="cri-o://324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91" gracePeriod=30 Nov 03 22:45:33.016754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.016725 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:33.113189 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.113090 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-xks-ca\") pod \"8a83b01c-bd97-4f4d-bb55-1db21191f756\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " Nov 03 22:45:33.113394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.113234 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a83b01c-bd97-4f4d-bb55-1db21191f756-kube-api-access-mqzln\" (UniqueName: \"kubernetes.io/projected/8a83b01c-bd97-4f4d-bb55-1db21191f756-kube-api-access-mqzln\") pod \"8a83b01c-bd97-4f4d-bb55-1db21191f756\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " Nov 03 22:45:33.113394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.113358 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-openshift-ca\") pod \"8a83b01c-bd97-4f4d-bb55-1db21191f756\" (UID: \"8a83b01c-bd97-4f4d-bb55-1db21191f756\") " Nov 03 22:45:33.113819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.113788 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-openshift-ca" pod "8a83b01c-bd97-4f4d-bb55-1db21191f756" (UID: "8a83b01c-bd97-4f4d-bb55-1db21191f756"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:33.114258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.114234 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:33.119342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.115626 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-xks-ca" pod "8a83b01c-bd97-4f4d-bb55-1db21191f756" (UID: "8a83b01c-bd97-4f4d-bb55-1db21191f756"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:33.119342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.115923 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a83b01c-bd97-4f4d-bb55-1db21191f756-kube-api-access-mqzln" pod "8a83b01c-bd97-4f4d-bb55-1db21191f756" (UID: "8a83b01c-bd97-4f4d-bb55-1db21191f756"). InnerVolumeSpecName "kube-api-access-mqzln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:33.215592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.215556 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqzln\" (UniqueName: \"kubernetes.io/projected/8a83b01c-bd97-4f4d-bb55-1db21191f756-kube-api-access-mqzln\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:33.215592 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.215591 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a83b01c-bd97-4f4d-bb55-1db21191f756-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:33.850772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.850743 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" Nov 03 22:45:33.850948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.850767 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7" event={"ID":"8a83b01c-bd97-4f4d-bb55-1db21191f756","Type":"ContainerDied","Data":"862af109432224536ea443e875c47e4cdecd8616dfad169bec5116ddf9aa2584"} Nov 03 22:45:33.850948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:33.850839 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862af109432224536ea443e875c47e4cdecd8616dfad169bec5116ddf9aa2584" Nov 03 22:45:34.460463 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.460432 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b"] Nov 03 22:45:34.461456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461434 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerName="cleanup" Nov 03 22:45:34.461537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461459 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerName="cleanup" Nov 03 22:45:34.461537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461481 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerName="cleanup" Nov 03 22:45:34.461537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461488 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63f0f29-03cf-4880-8e34-b44f25a1eb2a" containerName="cleanup" Nov 03 22:45:34.461537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461535 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461541 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9445c683-e8c3-41bf-9cbc-5d3093d840c6" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461555 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461561 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461574 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461580 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9725d5-d50c-496b-834e-a783e0c47b28" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461608 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461614 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461627 4903 cpu_manager.go:414] "containerMap: removing container" podUID="85615752-db5a-4dea-bc86-290e793928e0" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461632 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85615752-db5a-4dea-bc86-290e793928e0" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461646 4903 cpu_manager.go:414] "containerMap: removing container" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461652 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ad238a-db65-4160-a25b-9e29df9dbc94" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461662 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461668 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461679 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8a83b01c-bd97-4f4d-bb55-1db21191f756" containerName="revoke-keys" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461685 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a83b01c-bd97-4f4d-bb55-1db21191f756" containerName="revoke-keys" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461699 4903 cpu_manager.go:414] "containerMap: removing container" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" containerName="cleanup" Nov 03 22:45:34.461758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461705 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" containerName="cleanup" Nov 03 22:45:34.462568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461913 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb962f11-d246-41e8-a596-6383a2d91a58" containerName="cleanup" Nov 03 22:45:34.462568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461932 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="663b664e-98af-4583-aed0-da9572e83f52" containerName="cleanup" Nov 03 22:45:34.462568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461941 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d6c159-15e1-4fc7-8d50-40758bd126be" containerName="cleanup" Nov 03 22:45:34.462568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461951 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c71cca5-adfa-4f08-9233-2719cadc9550" containerName="cleanup" Nov 03 22:45:34.462568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461959 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a83b01c-bd97-4f4d-bb55-1db21191f756" containerName="revoke-keys" Nov 03 22:45:34.462568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461970 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d177e30a-9ef2-4d4d-a2be-3872a6bc9411" containerName="cleanup" Nov 03 22:45:34.462568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461983 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="659a4a6e-2d77-4d9f-8284-a29be2d2de56" containerName="cleanup" Nov 03 22:45:34.462568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.461992 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be9a4b9-2039-4d10-b1a5-f6a3cd3b2164" containerName="cleanup" Nov 03 22:45:34.462568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.462003 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b052a69f-e1a3-408e-9881-94770cb92ed6" containerName="cleanup" Nov 03 22:45:34.465975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.465950 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.486714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.486687 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b"] Nov 03 22:45:34.490366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.490342 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-341c51\"/\"default-dockercfg-6m4kv\"" Nov 03 22:45:34.490487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.490401 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-341c51\"/\"openshift-service-ca.crt\"" Nov 03 22:45:34.490487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.490412 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-341c51\"/\"kube-root-ca.crt\"" Nov 03 22:45:34.490590 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.490492 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-341c51\"/\"test-model-341c51-kserve-self-signed-certs\"" Nov 03 22:45:34.539660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.539626 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfxw\" (UniqueName: \"kubernetes.io/projected/09059d58-da66-4317-a7cf-a230f29333bc-kube-api-access-fqfxw\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.539860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.539668 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-tmp-dir\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.539860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.539763 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09059d58-da66-4317-a7cf-a230f29333bc-tls-certs\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.539860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.539822 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-home\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.540026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.540003 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-model-cache\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.540125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.540105 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-dshm\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.641957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.641919 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfxw\" (UniqueName: \"kubernetes.io/projected/09059d58-da66-4317-a7cf-a230f29333bc-kube-api-access-fqfxw\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.641957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.641962 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-tmp-dir\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.642187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.642006 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09059d58-da66-4317-a7cf-a230f29333bc-tls-certs\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.642187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.642059 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-home\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.642288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.642229 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-model-cache\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.642394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.642334 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-dshm\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.642583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.642555 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-tmp-dir\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.642691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.642585 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-home\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.642752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.642716 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-model-cache\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.644730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.644709 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-dshm\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.645036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.645015 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09059d58-da66-4317-a7cf-a230f29333bc-tls-certs\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.663560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.663536 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfxw\" (UniqueName: \"kubernetes.io/projected/09059d58-da66-4317-a7cf-a230f29333bc-kube-api-access-fqfxw\") pod \"test-model-341c51-kserve-6977dcb48f-rrv9b\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:34.795563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:34.794244 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:35.183347 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:35.183281 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09059d58_da66_4317_a7cf_a230f29333bc.slice/crio-ec85d264871fa910253bddd701c64154a45bc665a9c1f267c89f1af9cd4cebb0 WatchSource:0}: Error finding container ec85d264871fa910253bddd701c64154a45bc665a9c1f267c89f1af9cd4cebb0: Status 404 returned error can't find the container with id ec85d264871fa910253bddd701c64154a45bc665a9c1f267c89f1af9cd4cebb0 Nov 03 22:45:35.183552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:35.183531 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b"] Nov 03 22:45:35.894654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:35.894611 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" event={"ID":"09059d58-da66-4317-a7cf-a230f29333bc","Type":"ContainerStarted","Data":"546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7"} Nov 03 22:45:35.894654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:35.894659 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" event={"ID":"09059d58-da66-4317-a7cf-a230f29333bc","Type":"ContainerStarted","Data":"ec85d264871fa910253bddd701c64154a45bc665a9c1f267c89f1af9cd4cebb0"} Nov 03 22:45:35.940465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:35.940413 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" podStartSLOduration=1.940398947 podStartE2EDuration="1.940398947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:35.935268139 +0000 UTC m=+1962.339209114" watchObservedRunningTime="2026-11-03 22:45:35.940398947 +0000 UTC m=+1962.344339963" Nov 03 22:45:36.794787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:36.794756 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:37.621485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.621446 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 22:45:37.621998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.621506 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 22:45:37.776425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.776398 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:45:37.847479 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.847446 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4"] Nov 03 22:45:37.848641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.848611 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" Nov 03 22:45:37.848641 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.848640 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" Nov 03 22:45:37.848904 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.848879 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerName="istio-proxy" Nov 03 22:45:37.852033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.852014 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:37.869122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.866008 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-124490\"/\"default-dockercfg-xpjr5\"" Nov 03 22:45:37.869122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.866522 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-124490\"/\"kube-root-ca.crt\"" Nov 03 22:45:37.869122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.866652 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-124490\"/\"test-model-124490-kserve-self-signed-certs\"" Nov 03 22:45:37.869122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.868183 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-124490\"/\"openshift-service-ca.crt\"" Nov 03 22:45:37.870748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.870722 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4"] Nov 03 22:45:37.916401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.916368 4903 generic.go:356] "Generic (PLEG): container finished" podID="a1382513-d646-48b0-b0dd-9c525da84f0a" containerID="324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91" exitCode=0 Nov 03 22:45:37.916567 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.916515 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" Nov 03 22:45:37.916614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.916602 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" event={"ID":"a1382513-d646-48b0-b0dd-9c525da84f0a","Type":"ContainerDied","Data":"324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91"} Nov 03 22:45:37.916648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.916639 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5" event={"ID":"a1382513-d646-48b0-b0dd-9c525da84f0a","Type":"ContainerDied","Data":"6df293fa43416402c261f3545651cd298281b643cd5103eaed7c5559be4d1ea9"} Nov 03 22:45:37.916680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.916659 4903 scope.go:122] "RemoveContainer" containerID="324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91" Nov 03 22:45:37.920201 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920175 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1382513-d646-48b0-b0dd-9c525da84f0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1382513-d646-48b0-b0dd-9c525da84f0a-istiod-ca-cert\") pod \"a1382513-d646-48b0-b0dd-9c525da84f0a\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " Nov 03 22:45:37.920398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920256 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-credential-socket\") pod \"a1382513-d646-48b0-b0dd-9c525da84f0a\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " Nov 03 22:45:37.920398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920314 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-socket\") pod \"a1382513-d646-48b0-b0dd-9c525da84f0a\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " Nov 03 22:45:37.920398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920389 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-podinfo\") pod \"a1382513-d646-48b0-b0dd-9c525da84f0a\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " Nov 03 22:45:37.920601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920541 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-envoy\") pod \"a1382513-d646-48b0-b0dd-9c525da84f0a\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " Nov 03 22:45:37.920601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920555 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1382513-d646-48b0-b0dd-9c525da84f0a-istiod-ca-cert" pod "a1382513-d646-48b0-b0dd-9c525da84f0a" (UID: "a1382513-d646-48b0-b0dd-9c525da84f0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:45:37.920707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920618 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-kube-api-access-4zmhq\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-kube-api-access-4zmhq\") pod \"a1382513-d646-48b0-b0dd-9c525da84f0a\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " Nov 03 22:45:37.920760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920714 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-token\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-token\") pod \"a1382513-d646-48b0-b0dd-9c525da84f0a\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " Nov 03 22:45:37.920810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920798 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-data\") pod \"a1382513-d646-48b0-b0dd-9c525da84f0a\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " Nov 03 22:45:37.920873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920828 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-socket" pod "a1382513-d646-48b0-b0dd-9c525da84f0a" (UID: "a1382513-d646-48b0-b0dd-9c525da84f0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:37.920930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.920906 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-certs\") pod \"a1382513-d646-48b0-b0dd-9c525da84f0a\" (UID: \"a1382513-d646-48b0-b0dd-9c525da84f0a\") " Nov 03 22:45:37.921046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.921027 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-credential-socket" pod "a1382513-d646-48b0-b0dd-9c525da84f0a" (UID: "a1382513-d646-48b0-b0dd-9c525da84f0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:37.921929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.921904 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1382513-d646-48b0-b0dd-9c525da84f0a-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:37.922032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.921934 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:37.922032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.921948 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:37.922388 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.922364 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-data" pod "a1382513-d646-48b0-b0dd-9c525da84f0a" (UID: "a1382513-d646-48b0-b0dd-9c525da84f0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:37.922585 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.922562 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-certs" pod "a1382513-d646-48b0-b0dd-9c525da84f0a" (UID: "a1382513-d646-48b0-b0dd-9c525da84f0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:37.923244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.923216 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-envoy" pod "a1382513-d646-48b0-b0dd-9c525da84f0a" (UID: "a1382513-d646-48b0-b0dd-9c525da84f0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:37.923398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.923373 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-token" pod "a1382513-d646-48b0-b0dd-9c525da84f0a" (UID: "a1382513-d646-48b0-b0dd-9c525da84f0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:37.923836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.923811 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-podinfo" pod "a1382513-d646-48b0-b0dd-9c525da84f0a" (UID: "a1382513-d646-48b0-b0dd-9c525da84f0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:45:37.925436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.925416 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-kube-api-access-4zmhq" pod "a1382513-d646-48b0-b0dd-9c525da84f0a" (UID: "a1382513-d646-48b0-b0dd-9c525da84f0a"). InnerVolumeSpecName "kube-api-access-4zmhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:37.930245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.930228 4903 scope.go:122] "RemoveContainer" containerID="324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91" Nov 03 22:45:37.930555 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:37.930535 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91\": container with ID starting with 324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91 not found: ID does not exist" containerID="324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91" Nov 03 22:45:37.930612 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:37.930562 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91"} err="failed to get container status \"324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91\": rpc error: code = NotFound desc = could not find container \"324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91\": container with ID starting with 324d65ae7ec7cea9ec107ba2f7eb8412070cd4a11638dec93db7d2ba8f88fe91 not found: ID does not exist" Nov 03 22:45:38.023404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023273 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-home\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.023566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023431 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-model-cache\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.023566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023466 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wfm9\" (UniqueName: \"kubernetes.io/projected/e91da8b1-f0a1-439c-9643-078c90f8ea5e-kube-api-access-4wfm9\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.023657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023600 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-dshm\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.023743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023727 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tmp-dir\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.023788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023750 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tls-certs\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.023855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023844 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:38.023896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023881 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:38.023940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023901 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:38.023940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023911 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zmhq\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-kube-api-access-4zmhq\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:38.023940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023921 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:38.023940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.023930 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1382513-d646-48b0-b0dd-9c525da84f0a-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:38.130505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.127869 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tmp-dir\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.130505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.127923 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tls-certs\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.130505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.128108 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-home\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.130505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.128200 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-model-cache\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.130505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.128223 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfm9\" (UniqueName: \"kubernetes.io/projected/e91da8b1-f0a1-439c-9643-078c90f8ea5e-kube-api-access-4wfm9\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.130505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.128284 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-dshm\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.130505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.129244 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-model-cache\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.130505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.129516 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tmp-dir\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.130505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.130419 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-home\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.134464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.133316 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tls-certs\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.145069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.142685 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-dshm\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.160336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.160271 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfm9\" (UniqueName: \"kubernetes.io/projected/e91da8b1-f0a1-439c-9643-078c90f8ea5e-kube-api-access-4wfm9\") pod \"test-model-124490-kserve-6745d68f47-kxwp4\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.172278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.172239 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:38.279691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.279610 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5"] Nov 03 22:45:38.283917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.283827 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-fgvq5"] Nov 03 22:45:38.451482 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:38.451447 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:38.506513 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:45:38.506480 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode91da8b1_f0a1_439c_9643_078c90f8ea5e.slice/crio-004e90cd8175afa8def0d3531caefa9bbe8af4e3a1c19103feca7b84b1a988a8 WatchSource:0}: Error finding container 004e90cd8175afa8def0d3531caefa9bbe8af4e3a1c19103feca7b84b1a988a8: Status 404 returned error can't find the container with id 004e90cd8175afa8def0d3531caefa9bbe8af4e3a1c19103feca7b84b1a988a8 Nov 03 22:45:38.512257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.512226 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4"] Nov 03 22:45:38.806305 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:38.806211 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:38.909346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.906592 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7"] Nov 03 22:45:38.930633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.930601 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" event={"ID":"e91da8b1-f0a1-439c-9643-078c90f8ea5e","Type":"ContainerStarted","Data":"141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66"} Nov 03 22:45:38.930799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.930643 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" event={"ID":"e91da8b1-f0a1-439c-9643-078c90f8ea5e","Type":"ContainerStarted","Data":"004e90cd8175afa8def0d3531caefa9bbe8af4e3a1c19103feca7b84b1a988a8"} Nov 03 22:45:38.987459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:38.987401 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" podStartSLOduration=1.987384622 podStartE2EDuration="1.987384622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:45:38.983191425 +0000 UTC m=+1965.387132401" watchObservedRunningTime="2026-11-03 22:45:38.987384622 +0000 UTC m=+1965.391325628" Nov 03 22:45:39.172633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:39.172588 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:39.738261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:39.738226 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1382513-d646-48b0-b0dd-9c525da84f0a" path="/var/lib/kubelet/pods/a1382513-d646-48b0-b0dd-9c525da84f0a/volumes" Nov 03 22:45:40.160854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:40.160819 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw"] Nov 03 22:45:40.193189 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:40.193158 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h"] Nov 03 22:45:44.795153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:44.795116 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:44.808429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:44.808405 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:45.010155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:45.010126 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:45:45.161573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:45.161523 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" podUID="0cad264d-2123-47ad-846d-0f5762bc2b9f" containerName="payload-processing" containerID="cri-o://cb8464343bf30b4815c7fdf5947435175a58a82ec8ade42ec0e1edee9379e8e0" gracePeriod=25 Nov 03 22:45:45.194024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:45.193987 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" podUID="a60a004f-1b91-4d05-a295-3e61f2b88c35" containerName="payload-pre-processing" containerID="cri-o://dc44bcad55f566541f817847d2775953bc8097503e4c994169a052d8b66bb3b6" gracePeriod=25 Nov 03 22:45:48.138528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:48.138493 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:45:48.173130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:48.173082 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:48.186935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:48.186905 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:48.388950 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:48.388867 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:45:48.571544 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:48.571509 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:49.054942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:49.054913 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:45:51.646606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:51.646563 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" podUID="0cad264d-2123-47ad-846d-0f5762bc2b9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:45:51.647022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:51.646643 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" podUID="a60a004f-1b91-4d05-a295-3e61f2b88c35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:45:53.910582 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:45:53.910543 4903 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0239ba34_b4fd_44c7_9c54_c908dba5353e.slice/crio-8a0dee482560e8a3601e52d79afbe8bd6a862eae42fadd1fcc39c79a2f1301cf\": RecentStats: unable to find data in memory cache]" Nov 03 22:45:53.932128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:53.932086 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" podUID="10347389-d227-4891-9a5c-401dec199b97" containerName="main" containerID="cri-o://129e43adaf286f4b5119f5e5d984982881cb09266b17bd89826f2d74839f2884" gracePeriod=45 Nov 03 22:45:54.196245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.196158 4903 generic.go:356] "Generic (PLEG): container finished" podID="10347389-d227-4891-9a5c-401dec199b97" containerID="129e43adaf286f4b5119f5e5d984982881cb09266b17bd89826f2d74839f2884" exitCode=0 Nov 03 22:45:54.196245 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.196226 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" event={"ID":"10347389-d227-4891-9a5c-401dec199b97","Type":"ContainerDied","Data":"129e43adaf286f4b5119f5e5d984982881cb09266b17bd89826f2d74839f2884"} Nov 03 22:45:54.254971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.254938 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:45:54.297259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.297223 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10347389-d227-4891-9a5c-401dec199b97-kube-api-access-c7nrz\" (UniqueName: \"kubernetes.io/projected/10347389-d227-4891-9a5c-401dec199b97-kube-api-access-c7nrz\") pod \"10347389-d227-4891-9a5c-401dec199b97\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " Nov 03 22:45:54.297469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.297391 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-home\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-home\") pod \"10347389-d227-4891-9a5c-401dec199b97\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " Nov 03 22:45:54.297469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.297424 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-tmp-dir\") pod \"10347389-d227-4891-9a5c-401dec199b97\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " Nov 03 22:45:54.297593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.297499 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10347389-d227-4891-9a5c-401dec199b97-tls-certs\" (UniqueName: \"kubernetes.io/secret/10347389-d227-4891-9a5c-401dec199b97-tls-certs\") pod \"10347389-d227-4891-9a5c-401dec199b97\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " Nov 03 22:45:54.297660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.297638 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-dshm\") pod \"10347389-d227-4891-9a5c-401dec199b97\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " Nov 03 22:45:54.297717 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.297683 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-model-cache\") pod \"10347389-d227-4891-9a5c-401dec199b97\" (UID: \"10347389-d227-4891-9a5c-401dec199b97\") " Nov 03 22:45:54.297717 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.297698 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-tmp-dir" pod "10347389-d227-4891-9a5c-401dec199b97" (UID: "10347389-d227-4891-9a5c-401dec199b97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:54.297858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.297806 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-home" pod "10347389-d227-4891-9a5c-401dec199b97" (UID: "10347389-d227-4891-9a5c-401dec199b97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:54.297922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.297857 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-model-cache" pod "10347389-d227-4891-9a5c-401dec199b97" (UID: "10347389-d227-4891-9a5c-401dec199b97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:54.298429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.298405 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:54.298560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.298454 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:54.298560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.298470 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:54.299948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.299915 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10347389-d227-4891-9a5c-401dec199b97-kube-api-access-c7nrz" pod "10347389-d227-4891-9a5c-401dec199b97" (UID: "10347389-d227-4891-9a5c-401dec199b97"). InnerVolumeSpecName "kube-api-access-c7nrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:45:54.300228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.300200 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-dshm" pod "10347389-d227-4891-9a5c-401dec199b97" (UID: "10347389-d227-4891-9a5c-401dec199b97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:45:54.300419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.300395 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10347389-d227-4891-9a5c-401dec199b97-tls-certs" pod "10347389-d227-4891-9a5c-401dec199b97" (UID: "10347389-d227-4891-9a5c-401dec199b97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:45:54.399992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.399960 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10347389-d227-4891-9a5c-401dec199b97-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:54.399992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.399988 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10347389-d227-4891-9a5c-401dec199b97-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:54.399992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:54.399997 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7nrz\" (UniqueName: \"kubernetes.io/projected/10347389-d227-4891-9a5c-401dec199b97-kube-api-access-c7nrz\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:45:55.207077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:55.207046 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" Nov 03 22:45:55.207607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:55.207052 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7" event={"ID":"10347389-d227-4891-9a5c-401dec199b97","Type":"ContainerDied","Data":"070e7303b9bd417b9989a77f18420c4338b46434f222b8ac644d04a158073b9b"} Nov 03 22:45:55.207607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:55.207173 4903 scope.go:122] "RemoveContainer" containerID="129e43adaf286f4b5119f5e5d984982881cb09266b17bd89826f2d74839f2884" Nov 03 22:45:55.259362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:55.259330 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7"] Nov 03 22:45:55.269597 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:55.269565 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2d6076/test-model-2d6076-kserve-f99c557c-bdmv7"] Nov 03 22:45:55.737374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:55.737342 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10347389-d227-4891-9a5c-401dec199b97" path="/var/lib/kubelet/pods/10347389-d227-4891-9a5c-401dec199b97/volumes" Nov 03 22:45:58.310050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:58.310017 4903 scope.go:122] "RemoveContainer" containerID="91a5e29516f16afe8da000f51724828c5652344cdbe8fd1e2ddcd554df46a4d4" Nov 03 22:45:58.321929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:45:58.321906 4903 scope.go:122] "RemoveContainer" containerID="bacae7d091abd4b7c06df688fc64b55197fd58841cf696e2427fa99cd0281baa" Nov 03 22:46:01.646703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:01.646669 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" podUID="a60a004f-1b91-4d05-a295-3e61f2b88c35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:46:01.647115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:01.646728 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" podUID="0cad264d-2123-47ad-846d-0f5762bc2b9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:46:02.583782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:02.583746 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4"] Nov 03 22:46:02.583999 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:02.583978 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" containerID="cri-o://ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746" gracePeriod=30 Nov 03 22:46:05.330109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.330074 4903 generic.go:356] "Generic (PLEG): container finished" podID="0cad264d-2123-47ad-846d-0f5762bc2b9f" containerID="cb8464343bf30b4815c7fdf5947435175a58a82ec8ade42ec0e1edee9379e8e0" exitCode=0 Nov 03 22:46:05.330554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.330145 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" event={"ID":"0cad264d-2123-47ad-846d-0f5762bc2b9f","Type":"ContainerDied","Data":"cb8464343bf30b4815c7fdf5947435175a58a82ec8ade42ec0e1edee9379e8e0"} Nov 03 22:46:05.332161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.332137 4903 generic.go:356] "Generic (PLEG): container finished" podID="a60a004f-1b91-4d05-a295-3e61f2b88c35" containerID="dc44bcad55f566541f817847d2775953bc8097503e4c994169a052d8b66bb3b6" exitCode=0 Nov 03 22:46:05.332322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.332198 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" event={"ID":"a60a004f-1b91-4d05-a295-3e61f2b88c35","Type":"ContainerDied","Data":"dc44bcad55f566541f817847d2775953bc8097503e4c994169a052d8b66bb3b6"} Nov 03 22:46:05.605879 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.605855 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:46:05.609826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.609805 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:46:05.728522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.728489 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60a004f-1b91-4d05-a295-3e61f2b88c35-kube-api-access-9fb9c\" (UniqueName: \"kubernetes.io/projected/a60a004f-1b91-4d05-a295-3e61f2b88c35-kube-api-access-9fb9c\") pod \"a60a004f-1b91-4d05-a295-3e61f2b88c35\" (UID: \"a60a004f-1b91-4d05-a295-3e61f2b88c35\") " Nov 03 22:46:05.728709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.728550 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60a004f-1b91-4d05-a295-3e61f2b88c35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60a004f-1b91-4d05-a295-3e61f2b88c35-plugins-config-volume\") pod \"a60a004f-1b91-4d05-a295-3e61f2b88c35\" (UID: \"a60a004f-1b91-4d05-a295-3e61f2b88c35\") " Nov 03 22:46:05.728709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.728590 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cad264d-2123-47ad-846d-0f5762bc2b9f-kube-api-access-6tktl\" (UniqueName: \"kubernetes.io/projected/0cad264d-2123-47ad-846d-0f5762bc2b9f-kube-api-access-6tktl\") pod \"0cad264d-2123-47ad-846d-0f5762bc2b9f\" (UID: \"0cad264d-2123-47ad-846d-0f5762bc2b9f\") " Nov 03 22:46:05.728709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.728624 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cad264d-2123-47ad-846d-0f5762bc2b9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cad264d-2123-47ad-846d-0f5762bc2b9f-plugins-config-volume\") pod \"0cad264d-2123-47ad-846d-0f5762bc2b9f\" (UID: \"0cad264d-2123-47ad-846d-0f5762bc2b9f\") " Nov 03 22:46:05.729507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.729476 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cad264d-2123-47ad-846d-0f5762bc2b9f-plugins-config-volume" pod "0cad264d-2123-47ad-846d-0f5762bc2b9f" (UID: "0cad264d-2123-47ad-846d-0f5762bc2b9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:05.729705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.729647 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60a004f-1b91-4d05-a295-3e61f2b88c35-plugins-config-volume" pod "a60a004f-1b91-4d05-a295-3e61f2b88c35" (UID: "a60a004f-1b91-4d05-a295-3e61f2b88c35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:05.731928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.731902 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60a004f-1b91-4d05-a295-3e61f2b88c35-kube-api-access-9fb9c" pod "a60a004f-1b91-4d05-a295-3e61f2b88c35" (UID: "a60a004f-1b91-4d05-a295-3e61f2b88c35"). InnerVolumeSpecName "kube-api-access-9fb9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:05.732153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.732123 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cad264d-2123-47ad-846d-0f5762bc2b9f-kube-api-access-6tktl" pod "0cad264d-2123-47ad-846d-0f5762bc2b9f" (UID: "0cad264d-2123-47ad-846d-0f5762bc2b9f"). InnerVolumeSpecName "kube-api-access-6tktl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:05.831055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.830956 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60a004f-1b91-4d05-a295-3e61f2b88c35-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:05.831055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.830996 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tktl\" (UniqueName: \"kubernetes.io/projected/0cad264d-2123-47ad-846d-0f5762bc2b9f-kube-api-access-6tktl\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:05.831055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.831013 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cad264d-2123-47ad-846d-0f5762bc2b9f-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:05.831055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:05.831028 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fb9c\" (UniqueName: \"kubernetes.io/projected/a60a004f-1b91-4d05-a295-3e61f2b88c35-kube-api-access-9fb9c\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:06.341787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.341760 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" Nov 03 22:46:06.342256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.341779 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw" event={"ID":"0cad264d-2123-47ad-846d-0f5762bc2b9f","Type":"ContainerDied","Data":"b7a514154590e38653421d84b04ceb2f17be493d47bfb2432496c3ee25e3eb2b"} Nov 03 22:46:06.342256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.341846 4903 scope.go:122] "RemoveContainer" containerID="cb8464343bf30b4815c7fdf5947435175a58a82ec8ade42ec0e1edee9379e8e0" Nov 03 22:46:06.343490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.343468 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" Nov 03 22:46:06.343631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.343482 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h" event={"ID":"a60a004f-1b91-4d05-a295-3e61f2b88c35","Type":"ContainerDied","Data":"9c49c6acb6dbdf4d3503650c9a2ee20a791c4a7c25b7a7557e679a0520fc6842"} Nov 03 22:46:06.353640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.353623 4903 scope.go:122] "RemoveContainer" containerID="dc44bcad55f566541f817847d2775953bc8097503e4c994169a052d8b66bb3b6" Nov 03 22:46:06.399521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.399485 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h"] Nov 03 22:46:06.409824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.409794 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4b5e2b-7dd898677f-6768h"] Nov 03 22:46:06.433354 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.433322 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw"] Nov 03 22:46:06.437957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:06.437881 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4b5e2b-56d495d764-wxzjw"] Nov 03 22:46:07.740313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.740262 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cad264d-2123-47ad-846d-0f5762bc2b9f" path="/var/lib/kubelet/pods/0cad264d-2123-47ad-846d-0f5762bc2b9f/volumes" Nov 03 22:46:07.741038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.741016 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60a004f-1b91-4d05-a295-3e61f2b88c35" path="/var/lib/kubelet/pods/a60a004f-1b91-4d05-a295-3e61f2b88c35/volumes" Nov 03 22:46:07.893452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.893423 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:46:07.970363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.970285 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57055112-7228-4604-929c-ef621a1a85af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57055112-7228-4604-929c-ef621a1a85af-istiod-ca-cert\") pod \"57055112-7228-4604-929c-ef621a1a85af\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " Nov 03 22:46:07.970565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.970459 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-istio-token\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-istio-token\") pod \"57055112-7228-4604-929c-ef621a1a85af\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " Nov 03 22:46:07.970565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.970518 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-socket\") pod \"57055112-7228-4604-929c-ef621a1a85af\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " Nov 03 22:46:07.970707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.970616 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-data\") pod \"57055112-7228-4604-929c-ef621a1a85af\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " Nov 03 22:46:07.970707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.970656 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-kube-api-access-sgvmr\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-kube-api-access-sgvmr\") pod \"57055112-7228-4604-929c-ef621a1a85af\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " Nov 03 22:46:07.970707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.970682 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57055112-7228-4604-929c-ef621a1a85af-istiod-ca-cert" pod "57055112-7228-4604-929c-ef621a1a85af" (UID: "57055112-7228-4604-929c-ef621a1a85af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:07.970871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.970736 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-certs\") pod \"57055112-7228-4604-929c-ef621a1a85af\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " Nov 03 22:46:07.970871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.970839 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-envoy\") pod \"57055112-7228-4604-929c-ef621a1a85af\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " Nov 03 22:46:07.970985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.970872 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-credential-socket\") pod \"57055112-7228-4604-929c-ef621a1a85af\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " Nov 03 22:46:07.971038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.971004 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-data" pod "57055112-7228-4604-929c-ef621a1a85af" (UID: "57055112-7228-4604-929c-ef621a1a85af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:07.971038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.971016 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57055112-7228-4604-929c-ef621a1a85af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57055112-7228-4604-929c-ef621a1a85af-istio-podinfo\") pod \"57055112-7228-4604-929c-ef621a1a85af\" (UID: \"57055112-7228-4604-929c-ef621a1a85af\") " Nov 03 22:46:07.971038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.971017 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-socket" pod "57055112-7228-4604-929c-ef621a1a85af" (UID: "57055112-7228-4604-929c-ef621a1a85af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:07.971241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.971218 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-certs" pod "57055112-7228-4604-929c-ef621a1a85af" (UID: "57055112-7228-4604-929c-ef621a1a85af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:07.971775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.971747 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-credential-socket" pod "57055112-7228-4604-929c-ef621a1a85af" (UID: "57055112-7228-4604-929c-ef621a1a85af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:07.972208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.972189 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:07.972362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.972347 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57055112-7228-4604-929c-ef621a1a85af-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:07.972479 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.972466 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:07.972570 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.972557 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:07.972664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.972653 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:07.973876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.973849 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57055112-7228-4604-929c-ef621a1a85af-istio-podinfo" pod "57055112-7228-4604-929c-ef621a1a85af" (UID: "57055112-7228-4604-929c-ef621a1a85af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:46:07.974397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.974373 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-istio-token" pod "57055112-7228-4604-929c-ef621a1a85af" (UID: "57055112-7228-4604-929c-ef621a1a85af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:07.974769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.974745 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-envoy" pod "57055112-7228-4604-929c-ef621a1a85af" (UID: "57055112-7228-4604-929c-ef621a1a85af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:46:07.975088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:07.975068 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-kube-api-access-sgvmr" pod "57055112-7228-4604-929c-ef621a1a85af" (UID: "57055112-7228-4604-929c-ef621a1a85af"). InnerVolumeSpecName "kube-api-access-sgvmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:08.074728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.074691 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:08.074728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.074725 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvmr\" (UniqueName: \"kubernetes.io/projected/57055112-7228-4604-929c-ef621a1a85af-kube-api-access-sgvmr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:08.074728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.074737 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57055112-7228-4604-929c-ef621a1a85af-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:08.074982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.074751 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57055112-7228-4604-929c-ef621a1a85af-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:08.364253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.364214 4903 generic.go:356] "Generic (PLEG): container finished" podID="57055112-7228-4604-929c-ef621a1a85af" containerID="ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746" exitCode=0 Nov 03 22:46:08.364499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.364268 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" Nov 03 22:46:08.364499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.364335 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" event={"ID":"57055112-7228-4604-929c-ef621a1a85af","Type":"ContainerDied","Data":"ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746"} Nov 03 22:46:08.364499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.364404 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4" event={"ID":"57055112-7228-4604-929c-ef621a1a85af","Type":"ContainerDied","Data":"15cb2f7ca80fa6eed7f6d5aebb5e1ec70e7842c32e08650ff9ad2257a95d3755"} Nov 03 22:46:08.364499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.364434 4903 scope.go:122] "RemoveContainer" containerID="ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746" Nov 03 22:46:08.376948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.376926 4903 scope.go:122] "RemoveContainer" containerID="ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746" Nov 03 22:46:08.377332 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:46:08.377278 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746\": container with ID starting with ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746 not found: ID does not exist" containerID="ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746" Nov 03 22:46:08.377410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.377340 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746"} err="failed to get container status \"ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746\": rpc error: code = NotFound desc = could not find container \"ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746\": container with ID starting with ebe167f86201af13f71b7d5bc7d66b234c8c3ba03f2f8cde936102c3363ea746 not found: ID does not exist" Nov 03 22:46:08.415859 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.415830 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4"] Nov 03 22:46:08.425194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:08.425164 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-g45p4"] Nov 03 22:46:09.737800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:09.737763 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57055112-7228-4604-929c-ef621a1a85af" path="/var/lib/kubelet/pods/57055112-7228-4604-929c-ef621a1a85af/volumes" Nov 03 22:46:15.031499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:15.031463 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5"] Nov 03 22:46:15.041726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:15.041697 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0113354b-wp6f5"] Nov 03 22:46:15.737158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:15.737122 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1758bd64-657d-4cb3-a174-e28399527af7" path="/var/lib/kubelet/pods/1758bd64-657d-4cb3-a174-e28399527af7/volumes" Nov 03 22:46:18.442326 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:46:18.440510 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:46:21.954983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:21.954949 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7"] Nov 03 22:46:21.963792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:21.963766 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4b5e2b-6v2p7"] Nov 03 22:46:23.738146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:23.738104 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a83b01c-bd97-4f4d-bb55-1db21191f756" path="/var/lib/kubelet/pods/8a83b01c-bd97-4f4d-bb55-1db21191f756/volumes" Nov 03 22:46:30.871222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.871186 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk"] Nov 03 22:46:30.872146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872127 4903 cpu_manager.go:414] "containerMap: removing container" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" Nov 03 22:46:30.872146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872146 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" Nov 03 22:46:30.872253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872177 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a60a004f-1b91-4d05-a295-3e61f2b88c35" containerName="payload-pre-processing" Nov 03 22:46:30.872253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872183 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60a004f-1b91-4d05-a295-3e61f2b88c35" containerName="payload-pre-processing" Nov 03 22:46:30.872253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872236 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0cad264d-2123-47ad-846d-0f5762bc2b9f" containerName="payload-processing" Nov 03 22:46:30.872253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872245 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cad264d-2123-47ad-846d-0f5762bc2b9f" containerName="payload-processing" Nov 03 22:46:30.872501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872273 4903 cpu_manager.go:414] "containerMap: removing container" podUID="10347389-d227-4891-9a5c-401dec199b97" containerName="main" Nov 03 22:46:30.872501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872278 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10347389-d227-4891-9a5c-401dec199b97" containerName="main" Nov 03 22:46:30.872501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872476 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="10347389-d227-4891-9a5c-401dec199b97" containerName="main" Nov 03 22:46:30.872501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872495 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="57055112-7228-4604-929c-ef621a1a85af" containerName="istio-proxy" Nov 03 22:46:30.872634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872506 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60a004f-1b91-4d05-a295-3e61f2b88c35" containerName="payload-pre-processing" Nov 03 22:46:30.872634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.872519 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cad264d-2123-47ad-846d-0f5762bc2b9f" containerName="payload-processing" Nov 03 22:46:30.876497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.876480 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:30.902180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.902151 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk"] Nov 03 22:46:30.904823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.904803 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:46:30.904911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.904877 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:46:30.905408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.905388 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:46:30.905537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.905410 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:46:30.997500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.997458 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcqrn\" (UniqueName: \"kubernetes.io/projected/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-kube-api-access-gcqrn\") pod \"maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:30.997687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.997517 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:30.997687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:30.997646 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:31.098712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:31.098678 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:31.098999 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:31.098980 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcqrn\" (UniqueName: \"kubernetes.io/projected/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-kube-api-access-gcqrn\") pod \"maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:31.099089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:31.099015 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:31.099467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:31.099443 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:31.101469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:31.101444 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:31.121963 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:31.121888 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcqrn\" (UniqueName: \"kubernetes.io/projected/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-kube-api-access-gcqrn\") pod \"maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:31.192451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:31.192413 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:31.571182 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:31.571152 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk"] Nov 03 22:46:31.571820 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:46:31.571787 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82ecc33_cdc3_4270_a9c5_c1545dfe58fe.slice/crio-342fc137084e4d30bc1758952dd6c3bf4a4bbf85549b6ae164d5d6d2189d6bfe WatchSource:0}: Error finding container 342fc137084e4d30bc1758952dd6c3bf4a4bbf85549b6ae164d5d6d2189d6bfe: Status 404 returned error can't find the container with id 342fc137084e4d30bc1758952dd6c3bf4a4bbf85549b6ae164d5d6d2189d6bfe Nov 03 22:46:31.752600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:31.752556 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" event={"ID":"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe","Type":"ContainerStarted","Data":"342fc137084e4d30bc1758952dd6c3bf4a4bbf85549b6ae164d5d6d2189d6bfe"} Nov 03 22:46:32.764764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:32.764737 4903 generic.go:356] "Generic (PLEG): container finished" podID="b82ecc33-cdc3-4270-a9c5-c1545dfe58fe" containerID="cc203e7843861c0b7f42622c2ff6b6b54d184ad59f8e27b208c65b47578b96e7" exitCode=0 Nov 03 22:46:32.765101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:32.764818 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" event={"ID":"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe","Type":"ContainerDied","Data":"cc203e7843861c0b7f42622c2ff6b6b54d184ad59f8e27b208c65b47578b96e7"} Nov 03 22:46:33.992038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:33.992008 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:34.154646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.154621 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-kube-api-access-gcqrn\" (UniqueName: \"kubernetes.io/projected/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-kube-api-access-gcqrn\") pod \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " Nov 03 22:46:34.154815 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.154784 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-xks-ca\") pod \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " Nov 03 22:46:34.154878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.154865 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-openshift-ca\") pod \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\" (UID: \"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe\") " Nov 03 22:46:34.155304 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.155263 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-openshift-ca" pod "b82ecc33-cdc3-4270-a9c5-c1545dfe58fe" (UID: "b82ecc33-cdc3-4270-a9c5-c1545dfe58fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:34.155611 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.155592 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:34.156819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.156786 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-xks-ca" pod "b82ecc33-cdc3-4270-a9c5-c1545dfe58fe" (UID: "b82ecc33-cdc3-4270-a9c5-c1545dfe58fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:34.156920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.156854 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-kube-api-access-gcqrn" pod "b82ecc33-cdc3-4270-a9c5-c1545dfe58fe" (UID: "b82ecc33-cdc3-4270-a9c5-c1545dfe58fe"). InnerVolumeSpecName "kube-api-access-gcqrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:34.256953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.256916 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:34.256953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.256951 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcqrn\" (UniqueName: \"kubernetes.io/projected/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe-kube-api-access-gcqrn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:34.782837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.782808 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" Nov 03 22:46:34.783011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.782800 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk" event={"ID":"b82ecc33-cdc3-4270-a9c5-c1545dfe58fe","Type":"ContainerDied","Data":"342fc137084e4d30bc1758952dd6c3bf4a4bbf85549b6ae164d5d6d2189d6bfe"} Nov 03 22:46:34.783011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:34.782932 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="342fc137084e4d30bc1758952dd6c3bf4a4bbf85549b6ae164d5d6d2189d6bfe" Nov 03 22:46:41.318782 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:41.318700 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6"] Nov 03 22:46:46.319372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:46.319324 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" podUID="f562a97d-3a7d-476a-a5c6-9b102811d26b" containerName="payload-pre-processing" containerID="cri-o://2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9" gracePeriod=25 Nov 03 22:46:47.880661 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:47.880621 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" podUID="f562a97d-3a7d-476a-a5c6-9b102811d26b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:46:48.387540 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:46:48.387506 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:46:53.584886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.584849 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88"] Nov 03 22:46:53.587852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.587826 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b82ecc33-cdc3-4270-a9c5-c1545dfe58fe" containerName="revoke-keys" Nov 03 22:46:53.588012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.587998 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82ecc33-cdc3-4270-a9c5-c1545dfe58fe" containerName="revoke-keys" Nov 03 22:46:53.588632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.588613 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82ecc33-cdc3-4270-a9c5-c1545dfe58fe" containerName="revoke-keys" Nov 03 22:46:53.595158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.595135 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.604419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.604396 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88"] Nov 03 22:46:53.612667 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.612643 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:46:53.623104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.623078 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:46:53.623335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.623088 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:46:53.623505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.623183 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:46:53.715878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.715840 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9680aa83-sjk88\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.716070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.716047 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpn8s\" (UniqueName: \"kubernetes.io/projected/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-kube-api-access-wpn8s\") pod \"maas-api-revoke-sub-9680aa83-sjk88\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.716181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.716166 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9680aa83-sjk88\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.818434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.818398 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9680aa83-sjk88\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.818720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.818689 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpn8s\" (UniqueName: \"kubernetes.io/projected/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-kube-api-access-wpn8s\") pod \"maas-api-revoke-sub-9680aa83-sjk88\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.818988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.818966 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9680aa83-sjk88\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.830329 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.830307 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:46:53.831575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.831553 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:46:53.839970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.839912 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9680aa83-sjk88\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.841434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.841411 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9680aa83-sjk88\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.849632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.849612 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:46:53.861150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.861126 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpn8s\" (UniqueName: \"kubernetes.io/projected/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-kube-api-access-wpn8s\") pod \"maas-api-revoke-sub-9680aa83-sjk88\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:53.932531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.932507 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:46:53.940865 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:53.940845 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:54.283017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:54.282982 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88"] Nov 03 22:46:54.283275 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:46:54.283253 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2466f24f_1e0c_49d2_ba9e_dba2e8e3f967.slice/crio-9906d25c2c4a813035952a3499c257b04378d9cf2a939b9aeea1a8236d5ed2cf WatchSource:0}: Error finding container 9906d25c2c4a813035952a3499c257b04378d9cf2a939b9aeea1a8236d5ed2cf: Status 404 returned error can't find the container with id 9906d25c2c4a813035952a3499c257b04378d9cf2a939b9aeea1a8236d5ed2cf Nov 03 22:46:54.988940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:54.988906 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" event={"ID":"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967","Type":"ContainerStarted","Data":"dd322b70a566c8cb17cd27bf827160d8e57942fc69adeb9fd1cac5a2ad48a83f"} Nov 03 22:46:54.989387 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:54.988961 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" event={"ID":"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967","Type":"ContainerStarted","Data":"9906d25c2c4a813035952a3499c257b04378d9cf2a939b9aeea1a8236d5ed2cf"} Nov 03 22:46:55.028900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:55.028847 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" podStartSLOduration=2.0288332 podStartE2EDuration="2.0288332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:46:55.025060632 +0000 UTC m=+2041.429001607" watchObservedRunningTime="2026-11-03 22:46:55.0288332 +0000 UTC m=+2041.432774174" Nov 03 22:46:55.999231 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:55.999198 4903 generic.go:356] "Generic (PLEG): container finished" podID="2466f24f-1e0c-49d2-ba9e-dba2e8e3f967" containerID="dd322b70a566c8cb17cd27bf827160d8e57942fc69adeb9fd1cac5a2ad48a83f" exitCode=0 Nov 03 22:46:55.999775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:55.999363 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" event={"ID":"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967","Type":"ContainerDied","Data":"dd322b70a566c8cb17cd27bf827160d8e57942fc69adeb9fd1cac5a2ad48a83f"} Nov 03 22:46:57.232464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.232432 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:57.376437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.376322 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-kube-api-access-wpn8s\" (UniqueName: \"kubernetes.io/projected/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-kube-api-access-wpn8s\") pod \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " Nov 03 22:46:57.376663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.376444 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-openshift-ca\") pod \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " Nov 03 22:46:57.376663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.376516 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-xks-ca\") pod \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\" (UID: \"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967\") " Nov 03 22:46:57.377227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.377187 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-openshift-ca" pod "2466f24f-1e0c-49d2-ba9e-dba2e8e3f967" (UID: "2466f24f-1e0c-49d2-ba9e-dba2e8e3f967"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:46:57.388631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.388586 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-kube-api-access-wpn8s" pod "2466f24f-1e0c-49d2-ba9e-dba2e8e3f967" (UID: "2466f24f-1e0c-49d2-ba9e-dba2e8e3f967"). InnerVolumeSpecName "kube-api-access-wpn8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:46:57.390043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.390008 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-xks-ca" pod "2466f24f-1e0c-49d2-ba9e-dba2e8e3f967" (UID: "2466f24f-1e0c-49d2-ba9e-dba2e8e3f967"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:46:57.478791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.478752 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpn8s\" (UniqueName: \"kubernetes.io/projected/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-kube-api-access-wpn8s\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:57.478791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.478790 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:57.479010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.478806 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:46:57.880087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:57.880052 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" podUID="f562a97d-3a7d-476a-a5c6-9b102811d26b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:46:58.017615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:58.017534 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" Nov 03 22:46:58.017615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:58.017554 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88" event={"ID":"2466f24f-1e0c-49d2-ba9e-dba2e8e3f967","Type":"ContainerDied","Data":"9906d25c2c4a813035952a3499c257b04378d9cf2a939b9aeea1a8236d5ed2cf"} Nov 03 22:46:58.017615 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:58.017604 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9906d25c2c4a813035952a3499c257b04378d9cf2a939b9aeea1a8236d5ed2cf" Nov 03 22:46:58.489494 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:46:58.489459 4903 scope.go:122] "RemoveContainer" containerID="75948a0ec88ca19a7595ee9194708583006378b41a0e9d835a863b18f487960e" Nov 03 22:47:05.150816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.150777 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm"] Nov 03 22:47:05.152084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.152063 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2466f24f-1e0c-49d2-ba9e-dba2e8e3f967" containerName="revoke-keys" Nov 03 22:47:05.152158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.152087 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2466f24f-1e0c-49d2-ba9e-dba2e8e3f967" containerName="revoke-keys" Nov 03 22:47:05.152415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.152400 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2466f24f-1e0c-49d2-ba9e-dba2e8e3f967" containerName="revoke-keys" Nov 03 22:47:05.158001 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.157977 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.171910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.171885 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm"] Nov 03 22:47:05.174556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.174534 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:47:05.174697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.174603 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:47:05.174697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.174620 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:47:05.182129 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.181243 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:47:05.228335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.216345 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1802c6fa-lvtjm\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.228335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.216506 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1802c6fa-lvtjm\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.228335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.216617 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57wv\" (UniqueName: \"kubernetes.io/projected/416cc0c8-1592-479d-ba47-cc57da5059a1-kube-api-access-w57wv\") pod \"maas-api-revoke-sub-1802c6fa-lvtjm\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.318202 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.318160 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57wv\" (UniqueName: \"kubernetes.io/projected/416cc0c8-1592-479d-ba47-cc57da5059a1-kube-api-access-w57wv\") pod \"maas-api-revoke-sub-1802c6fa-lvtjm\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.318679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.318657 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1802c6fa-lvtjm\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.318970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.318943 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1802c6fa-lvtjm\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.319769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.319739 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1802c6fa-lvtjm\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.321691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.321670 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1802c6fa-lvtjm\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.340992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.340958 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57wv\" (UniqueName: \"kubernetes.io/projected/416cc0c8-1592-479d-ba47-cc57da5059a1-kube-api-access-w57wv\") pod \"maas-api-revoke-sub-1802c6fa-lvtjm\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.476191 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.476111 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:05.805839 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:05.805809 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm"] Nov 03 22:47:05.806141 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:47:05.806117 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416cc0c8_1592_479d_ba47_cc57da5059a1.slice/crio-723e6558086f930249b0914f387524a512e5edfc5c3ebba6f519974e34c17eaf WatchSource:0}: Error finding container 723e6558086f930249b0914f387524a512e5edfc5c3ebba6f519974e34c17eaf: Status 404 returned error can't find the container with id 723e6558086f930249b0914f387524a512e5edfc5c3ebba6f519974e34c17eaf Nov 03 22:47:06.090606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:06.090573 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" event={"ID":"416cc0c8-1592-479d-ba47-cc57da5059a1","Type":"ContainerStarted","Data":"723e6558086f930249b0914f387524a512e5edfc5c3ebba6f519974e34c17eaf"} Nov 03 22:47:06.640150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:06.640123 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:47:06.742403 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:06.742370 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f562a97d-3a7d-476a-a5c6-9b102811d26b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f562a97d-3a7d-476a-a5c6-9b102811d26b-plugins-config-volume\") pod \"f562a97d-3a7d-476a-a5c6-9b102811d26b\" (UID: \"f562a97d-3a7d-476a-a5c6-9b102811d26b\") " Nov 03 22:47:06.742610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:06.742562 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f562a97d-3a7d-476a-a5c6-9b102811d26b-kube-api-access-jmqjx\" (UniqueName: \"kubernetes.io/projected/f562a97d-3a7d-476a-a5c6-9b102811d26b-kube-api-access-jmqjx\") pod \"f562a97d-3a7d-476a-a5c6-9b102811d26b\" (UID: \"f562a97d-3a7d-476a-a5c6-9b102811d26b\") " Nov 03 22:47:06.742871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:06.742842 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f562a97d-3a7d-476a-a5c6-9b102811d26b-plugins-config-volume" pod "f562a97d-3a7d-476a-a5c6-9b102811d26b" (UID: "f562a97d-3a7d-476a-a5c6-9b102811d26b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:06.743398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:06.743375 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f562a97d-3a7d-476a-a5c6-9b102811d26b-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:06.745458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:06.745057 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f562a97d-3a7d-476a-a5c6-9b102811d26b-kube-api-access-jmqjx" pod "f562a97d-3a7d-476a-a5c6-9b102811d26b" (UID: "f562a97d-3a7d-476a-a5c6-9b102811d26b"). InnerVolumeSpecName "kube-api-access-jmqjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:06.844986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:06.844953 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmqjx\" (UniqueName: \"kubernetes.io/projected/f562a97d-3a7d-476a-a5c6-9b102811d26b-kube-api-access-jmqjx\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:07.101602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.101567 4903 generic.go:356] "Generic (PLEG): container finished" podID="f562a97d-3a7d-476a-a5c6-9b102811d26b" containerID="2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9" exitCode=0 Nov 03 22:47:07.101783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.101612 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" Nov 03 22:47:07.101783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.101649 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" event={"ID":"f562a97d-3a7d-476a-a5c6-9b102811d26b","Type":"ContainerDied","Data":"2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9"} Nov 03 22:47:07.101783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.101706 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6" event={"ID":"f562a97d-3a7d-476a-a5c6-9b102811d26b","Type":"ContainerDied","Data":"9f1d1cc0494e13fa43e30e93f6f5c33873bfeaeea358d2204bc76da5b28c8fbd"} Nov 03 22:47:07.101783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.101721 4903 scope.go:122] "RemoveContainer" containerID="2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9" Nov 03 22:47:07.103530 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.103505 4903 generic.go:356] "Generic (PLEG): container finished" podID="416cc0c8-1592-479d-ba47-cc57da5059a1" containerID="812fb1afdd7d9105107594a85b272d036322ac6c3e3fbd35b110ee16e8396815" exitCode=0 Nov 03 22:47:07.103662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.103567 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" event={"ID":"416cc0c8-1592-479d-ba47-cc57da5059a1","Type":"ContainerDied","Data":"812fb1afdd7d9105107594a85b272d036322ac6c3e3fbd35b110ee16e8396815"} Nov 03 22:47:07.114031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.113995 4903 scope.go:122] "RemoveContainer" containerID="2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9" Nov 03 22:47:07.114344 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:47:07.114288 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9\": container with ID starting with 2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9 not found: ID does not exist" containerID="2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9" Nov 03 22:47:07.114464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.114350 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9"} err="failed to get container status \"2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9\": rpc error: code = NotFound desc = could not find container \"2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9\": container with ID starting with 2d441c978cce27f9dfa8ed2faebcb0da0d3b9654861ee7445dcf8d704e3cd3e9 not found: ID does not exist" Nov 03 22:47:07.151101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.151064 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6"] Nov 03 22:47:07.163756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.163724 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2d6076-dc5c6ff87-4d8d6"] Nov 03 22:47:07.738539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:07.738501 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f562a97d-3a7d-476a-a5c6-9b102811d26b" path="/var/lib/kubelet/pods/f562a97d-3a7d-476a-a5c6-9b102811d26b/volumes" Nov 03 22:47:08.332740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.332712 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:08.473432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.473395 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416cc0c8-1592-479d-ba47-cc57da5059a1-kube-api-access-w57wv\" (UniqueName: \"kubernetes.io/projected/416cc0c8-1592-479d-ba47-cc57da5059a1-kube-api-access-w57wv\") pod \"416cc0c8-1592-479d-ba47-cc57da5059a1\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " Nov 03 22:47:08.474702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.474004 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-xks-ca\") pod \"416cc0c8-1592-479d-ba47-cc57da5059a1\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " Nov 03 22:47:08.474702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.474097 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-openshift-ca\") pod \"416cc0c8-1592-479d-ba47-cc57da5059a1\" (UID: \"416cc0c8-1592-479d-ba47-cc57da5059a1\") " Nov 03 22:47:08.475970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.475626 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-openshift-ca" pod "416cc0c8-1592-479d-ba47-cc57da5059a1" (UID: "416cc0c8-1592-479d-ba47-cc57da5059a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:08.479449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.478287 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-xks-ca" pod "416cc0c8-1592-479d-ba47-cc57da5059a1" (UID: "416cc0c8-1592-479d-ba47-cc57da5059a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:08.481765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.481737 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416cc0c8-1592-479d-ba47-cc57da5059a1-kube-api-access-w57wv" pod "416cc0c8-1592-479d-ba47-cc57da5059a1" (UID: "416cc0c8-1592-479d-ba47-cc57da5059a1"). InnerVolumeSpecName "kube-api-access-w57wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:08.576978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.576932 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:08.576978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.576975 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w57wv\" (UniqueName: \"kubernetes.io/projected/416cc0c8-1592-479d-ba47-cc57da5059a1-kube-api-access-w57wv\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:08.576978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:08.576985 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416cc0c8-1592-479d-ba47-cc57da5059a1-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:09.123849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:09.123818 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" event={"ID":"416cc0c8-1592-479d-ba47-cc57da5059a1","Type":"ContainerDied","Data":"723e6558086f930249b0914f387524a512e5edfc5c3ebba6f519974e34c17eaf"} Nov 03 22:47:09.124349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:09.123872 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="723e6558086f930249b0914f387524a512e5edfc5c3ebba6f519974e34c17eaf" Nov 03 22:47:09.124349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:09.123828 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm" Nov 03 22:47:18.387264 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:47:18.387228 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:47:19.546397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:19.546359 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk"] Nov 03 22:47:19.551747 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:19.551714 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2d6076-6zplk"] Nov 03 22:47:19.745804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:19.745764 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82ecc33-cdc3-4270-a9c5-c1545dfe58fe" path="/var/lib/kubelet/pods/b82ecc33-cdc3-4270-a9c5-c1545dfe58fe/volumes" Nov 03 22:47:25.522932 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:25.522892 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:47:34.226422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:34.226386 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9"] Nov 03 22:47:34.226874 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:34.226600 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" containerID="cri-o://e424f79d1bf43a378361a952cf40f646f82fea5ced2cc7b142e432035df7d97f" gracePeriod=30 Nov 03 22:47:35.814019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.813987 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc"] Nov 03 22:47:35.814924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.814905 4903 cpu_manager.go:414] "containerMap: removing container" podUID="416cc0c8-1592-479d-ba47-cc57da5059a1" containerName="revoke-keys" Nov 03 22:47:35.814924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.814924 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416cc0c8-1592-479d-ba47-cc57da5059a1" containerName="revoke-keys" Nov 03 22:47:35.815020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.814991 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f562a97d-3a7d-476a-a5c6-9b102811d26b" containerName="payload-pre-processing" Nov 03 22:47:35.815020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.814997 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f562a97d-3a7d-476a-a5c6-9b102811d26b" containerName="payload-pre-processing" Nov 03 22:47:35.815194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.815174 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f562a97d-3a7d-476a-a5c6-9b102811d26b" containerName="payload-pre-processing" Nov 03 22:47:35.815340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.815202 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="416cc0c8-1592-479d-ba47-cc57da5059a1" containerName="revoke-keys" Nov 03 22:47:35.820840 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.820819 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:35.836180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.836155 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc"] Nov 03 22:47:35.840354 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.840323 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d4b6bb-dockercfg-fddmr\"" Nov 03 22:47:35.840487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.840418 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d4b6bb\"" Nov 03 22:47:35.877966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.877922 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6f5\" (UniqueName: \"kubernetes.io/projected/17e621b5-3a21-4a9d-b192-dc3c126578a6-kube-api-access-bc6f5\") pod \"payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc\" (UID: \"17e621b5-3a21-4a9d-b192-dc3c126578a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:35.878199 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.878174 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e621b5-3a21-4a9d-b192-dc3c126578a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc\" (UID: \"17e621b5-3a21-4a9d-b192-dc3c126578a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:35.983533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.983458 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6f5\" (UniqueName: \"kubernetes.io/projected/17e621b5-3a21-4a9d-b192-dc3c126578a6-kube-api-access-bc6f5\") pod \"payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc\" (UID: \"17e621b5-3a21-4a9d-b192-dc3c126578a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:35.984102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.984055 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e621b5-3a21-4a9d-b192-dc3c126578a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc\" (UID: \"17e621b5-3a21-4a9d-b192-dc3c126578a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:35.985460 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:35.985405 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e621b5-3a21-4a9d-b192-dc3c126578a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc\" (UID: \"17e621b5-3a21-4a9d-b192-dc3c126578a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:36.063324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:36.063237 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6f5\" (UniqueName: \"kubernetes.io/projected/17e621b5-3a21-4a9d-b192-dc3c126578a6-kube-api-access-bc6f5\") pod \"payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc\" (UID: \"17e621b5-3a21-4a9d-b192-dc3c126578a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:36.140006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:36.139918 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:36.451413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:36.451388 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc"] Nov 03 22:47:36.451554 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:47:36.451390 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e621b5_3a21_4a9d_b192_dc3c126578a6.slice/crio-0bba0044f7507e593343ffb6c6377562d41561f6ca53112fc378a56d67f42782 WatchSource:0}: Error finding container 0bba0044f7507e593343ffb6c6377562d41561f6ca53112fc378a56d67f42782: Status 404 returned error can't find the container with id 0bba0044f7507e593343ffb6c6377562d41561f6ca53112fc378a56d67f42782 Nov 03 22:47:37.452167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:37.452124 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" event={"ID":"17e621b5-3a21-4a9d-b192-dc3c126578a6","Type":"ContainerStarted","Data":"5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0"} Nov 03 22:47:37.452675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:37.452215 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" event={"ID":"17e621b5-3a21-4a9d-b192-dc3c126578a6","Type":"ContainerStarted","Data":"0bba0044f7507e593343ffb6c6377562d41561f6ca53112fc378a56d67f42782"} Nov 03 22:47:37.452675 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:37.452239 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:37.492140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:37.492079 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" podStartSLOduration=2.492065052 podStartE2EDuration="2.492065052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:37.487463587 +0000 UTC m=+2083.891404543" watchObservedRunningTime="2026-11-03 22:47:37.492065052 +0000 UTC m=+2083.896006028" Nov 03 22:47:38.772151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:38.772108 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4"] Nov 03 22:47:39.067206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.067108 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx"] Nov 03 22:47:39.072764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.072738 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.085850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.085821 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:47:39.086169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.086128 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:47:39.092346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.092317 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx"] Nov 03 22:47:39.095851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.095828 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:47:39.095976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.095881 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:47:39.160869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.160836 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.161083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.160878 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fhl8\" (UniqueName: \"kubernetes.io/projected/49a41773-99db-42c3-99c3-aa5f834f8682-kube-api-access-2fhl8\") pod \"maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.161197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.161177 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.262053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.262024 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.262208 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.262189 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.262263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.262238 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fhl8\" (UniqueName: \"kubernetes.io/projected/49a41773-99db-42c3-99c3-aa5f834f8682-kube-api-access-2fhl8\") pod \"maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.263104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.263076 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.265472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.265451 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.286276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.286238 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fhl8\" (UniqueName: \"kubernetes.io/projected/49a41773-99db-42c3-99c3-aa5f834f8682-kube-api-access-2fhl8\") pod \"maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.388378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.388347 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:39.483326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.483253 4903 generic.go:356] "Generic (PLEG): container finished" podID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerID="e424f79d1bf43a378361a952cf40f646f82fea5ced2cc7b142e432035df7d97f" exitCode=0 Nov 03 22:47:39.483568 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.483508 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" event={"ID":"6d4e44ab-74c0-48d1-8d22-4c9d683a077a","Type":"ContainerDied","Data":"e424f79d1bf43a378361a952cf40f646f82fea5ced2cc7b142e432035df7d97f"} Nov 03 22:47:39.537066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.537032 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:47:39.668480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.668448 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istiod-ca-cert\") pod \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " Nov 03 22:47:39.668684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.668619 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-podinfo\") pod \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " Nov 03 22:47:39.668684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.668652 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-envoy\") pod \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " Nov 03 22:47:39.668888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.668700 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-certs\") pod \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " Nov 03 22:47:39.668888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.668737 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-socket\") pod \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " Nov 03 22:47:39.668888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.668763 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-data\") pod \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " Nov 03 22:47:39.668888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.668816 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-token\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-token\") pod \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " Nov 03 22:47:39.669161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.668901 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istiod-ca-cert" pod "6d4e44ab-74c0-48d1-8d22-4c9d683a077a" (UID: "6d4e44ab-74c0-48d1-8d22-4c9d683a077a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:39.669161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669009 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-kube-api-access-wmvxg\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-kube-api-access-wmvxg\") pod \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " Nov 03 22:47:39.669161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669042 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-certs" pod "6d4e44ab-74c0-48d1-8d22-4c9d683a077a" (UID: "6d4e44ab-74c0-48d1-8d22-4c9d683a077a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:39.669161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669095 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-credential-socket\") pod \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\" (UID: \"6d4e44ab-74c0-48d1-8d22-4c9d683a077a\") " Nov 03 22:47:39.669462 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669187 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-socket" pod "6d4e44ab-74c0-48d1-8d22-4c9d683a077a" (UID: "6d4e44ab-74c0-48d1-8d22-4c9d683a077a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:39.669462 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669266 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-data" pod "6d4e44ab-74c0-48d1-8d22-4c9d683a077a" (UID: "6d4e44ab-74c0-48d1-8d22-4c9d683a077a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:39.669587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669541 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-credential-socket" pod "6d4e44ab-74c0-48d1-8d22-4c9d683a077a" (UID: "6d4e44ab-74c0-48d1-8d22-4c9d683a077a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:39.669961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669936 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:39.670067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669966 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:39.670067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669982 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:39.670067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.669995 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:39.670067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.670010 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:39.671684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.671653 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-kube-api-access-wmvxg" pod "6d4e44ab-74c0-48d1-8d22-4c9d683a077a" (UID: "6d4e44ab-74c0-48d1-8d22-4c9d683a077a"). InnerVolumeSpecName "kube-api-access-wmvxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:39.671887 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.671865 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-envoy" pod "6d4e44ab-74c0-48d1-8d22-4c9d683a077a" (UID: "6d4e44ab-74c0-48d1-8d22-4c9d683a077a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:39.672183 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.672165 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-token" pod "6d4e44ab-74c0-48d1-8d22-4c9d683a077a" (UID: "6d4e44ab-74c0-48d1-8d22-4c9d683a077a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:39.672490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.672458 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-podinfo" pod "6d4e44ab-74c0-48d1-8d22-4c9d683a077a" (UID: "6d4e44ab-74c0-48d1-8d22-4c9d683a077a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:47:39.769550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.769520 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx"] Nov 03 22:47:39.771915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.771883 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:39.771915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.771907 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:39.772094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.771922 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:39.772094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:39.771931 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmvxg\" (UniqueName: \"kubernetes.io/projected/6d4e44ab-74c0-48d1-8d22-4c9d683a077a-kube-api-access-wmvxg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:40.493469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:40.493429 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" event={"ID":"49a41773-99db-42c3-99c3-aa5f834f8682","Type":"ContainerStarted","Data":"4f8fe78fb152079d45d8a8a82b2ba88fc6bbeee0bba592d9e9e736b35629097b"} Nov 03 22:47:40.493921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:40.493505 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" event={"ID":"49a41773-99db-42c3-99c3-aa5f834f8682","Type":"ContainerStarted","Data":"d5d8068175beff75e36ac306df34150878d4d0340c6408f39fba2fb9e258754e"} Nov 03 22:47:40.494935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:40.494915 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" Nov 03 22:47:40.495042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:40.494943 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9" event={"ID":"6d4e44ab-74c0-48d1-8d22-4c9d683a077a","Type":"ContainerDied","Data":"ec005ba060531a0ce177325704aa602ebd8fc3b70b3d10beb4473078c0b599e2"} Nov 03 22:47:40.495042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:40.494994 4903 scope.go:122] "RemoveContainer" containerID="e424f79d1bf43a378361a952cf40f646f82fea5ced2cc7b142e432035df7d97f" Nov 03 22:47:40.532203 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:40.532114 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" podStartSLOduration=1.532100782 podStartE2EDuration="1.532100782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:40.529374979 +0000 UTC m=+2086.933315952" watchObservedRunningTime="2026-11-03 22:47:40.532100782 +0000 UTC m=+2086.936041757" Nov 03 22:47:40.565899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:40.565863 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9"] Nov 03 22:47:40.575030 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:40.575008 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-qhqx9"] Nov 03 22:47:41.506079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:41.506048 4903 generic.go:356] "Generic (PLEG): container finished" podID="49a41773-99db-42c3-99c3-aa5f834f8682" containerID="4f8fe78fb152079d45d8a8a82b2ba88fc6bbeee0bba592d9e9e736b35629097b" exitCode=0 Nov 03 22:47:41.506583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:41.506131 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" event={"ID":"49a41773-99db-42c3-99c3-aa5f834f8682","Type":"ContainerDied","Data":"4f8fe78fb152079d45d8a8a82b2ba88fc6bbeee0bba592d9e9e736b35629097b"} Nov 03 22:47:41.741416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:41.740239 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" path="/var/lib/kubelet/pods/6d4e44ab-74c0-48d1-8d22-4c9d683a077a/volumes" Nov 03 22:47:42.687805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.687780 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:42.825465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.825379 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a41773-99db-42c3-99c3-aa5f834f8682-kube-api-access-2fhl8\" (UniqueName: \"kubernetes.io/projected/49a41773-99db-42c3-99c3-aa5f834f8682-kube-api-access-2fhl8\") pod \"49a41773-99db-42c3-99c3-aa5f834f8682\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " Nov 03 22:47:42.825660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.825607 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-xks-ca\") pod \"49a41773-99db-42c3-99c3-aa5f834f8682\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " Nov 03 22:47:42.825660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.825642 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-openshift-ca\") pod \"49a41773-99db-42c3-99c3-aa5f834f8682\" (UID: \"49a41773-99db-42c3-99c3-aa5f834f8682\") " Nov 03 22:47:42.829014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.826753 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-openshift-ca" pod "49a41773-99db-42c3-99c3-aa5f834f8682" (UID: "49a41773-99db-42c3-99c3-aa5f834f8682"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:42.829014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.827843 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:42.829454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.829424 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a41773-99db-42c3-99c3-aa5f834f8682-kube-api-access-2fhl8" pod "49a41773-99db-42c3-99c3-aa5f834f8682" (UID: "49a41773-99db-42c3-99c3-aa5f834f8682"). InnerVolumeSpecName "kube-api-access-2fhl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:42.843823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.843793 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-xks-ca" pod "49a41773-99db-42c3-99c3-aa5f834f8682" (UID: "49a41773-99db-42c3-99c3-aa5f834f8682"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:42.929777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.929742 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a41773-99db-42c3-99c3-aa5f834f8682-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:42.929777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:42.929774 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fhl8\" (UniqueName: \"kubernetes.io/projected/49a41773-99db-42c3-99c3-aa5f834f8682-kube-api-access-2fhl8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:43.541746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:43.541715 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" Nov 03 22:47:43.541746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:43.541729 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx" event={"ID":"49a41773-99db-42c3-99c3-aa5f834f8682","Type":"ContainerDied","Data":"d5d8068175beff75e36ac306df34150878d4d0340c6408f39fba2fb9e258754e"} Nov 03 22:47:43.541991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:43.541843 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5d8068175beff75e36ac306df34150878d4d0340c6408f39fba2fb9e258754e" Nov 03 22:47:48.386780 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:47:48.386737 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:47:48.466584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:48.466555 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 22:47:49.467722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:49.467690 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm"] Nov 03 22:47:49.510116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:49.510083 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx"] Nov 03 22:47:51.406585 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.406549 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59"] Nov 03 22:47:51.407429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.407409 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" Nov 03 22:47:51.407488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.407431 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" Nov 03 22:47:51.407488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.407449 4903 cpu_manager.go:414] "containerMap: removing container" podUID="49a41773-99db-42c3-99c3-aa5f834f8682" containerName="revoke-keys" Nov 03 22:47:51.407488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.407454 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a41773-99db-42c3-99c3-aa5f834f8682" containerName="revoke-keys" Nov 03 22:47:51.407644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.407634 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4e44ab-74c0-48d1-8d22-4c9d683a077a" containerName="istio-proxy" Nov 03 22:47:51.407681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.407651 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a41773-99db-42c3-99c3-aa5f834f8682" containerName="revoke-keys" Nov 03 22:47:51.411539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.411522 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.428453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.428430 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59"] Nov 03 22:47:51.432857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.432834 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:47:51.432857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.432834 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:47:51.433019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.432860 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:47:51.433019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.432888 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:47:51.553734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.553705 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5wq\" (UniqueName: \"kubernetes.io/projected/a68a75c6-1469-4ce8-b300-e010e6b48b19-kube-api-access-5v5wq\") pod \"maas-api-revoke-sub-4865466c-mqs59\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.553914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.553886 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4865466c-mqs59\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.554080 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.554059 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4865466c-mqs59\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.655720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.655682 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wq\" (UniqueName: \"kubernetes.io/projected/a68a75c6-1469-4ce8-b300-e010e6b48b19-kube-api-access-5v5wq\") pod \"maas-api-revoke-sub-4865466c-mqs59\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.655928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.655853 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4865466c-mqs59\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.656032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.656016 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4865466c-mqs59\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.656682 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.656628 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4865466c-mqs59\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.658146 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.658130 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4865466c-mqs59\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.677015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.676991 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5wq\" (UniqueName: \"kubernetes.io/projected/a68a75c6-1469-4ce8-b300-e010e6b48b19-kube-api-access-5v5wq\") pod \"maas-api-revoke-sub-4865466c-mqs59\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:51.727998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:51.727971 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:52.045112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:52.045069 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59"] Nov 03 22:47:52.045624 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:47:52.045598 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68a75c6_1469_4ce8_b300_e010e6b48b19.slice/crio-718a6e36be0aa121e7206ae867caec6d467e14cfe1ee6957cf16ba41c16a9d82 WatchSource:0}: Error finding container 718a6e36be0aa121e7206ae867caec6d467e14cfe1ee6957cf16ba41c16a9d82: Status 404 returned error can't find the container with id 718a6e36be0aa121e7206ae867caec6d467e14cfe1ee6957cf16ba41c16a9d82 Nov 03 22:47:52.645795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:52.645701 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" event={"ID":"a68a75c6-1469-4ce8-b300-e010e6b48b19","Type":"ContainerStarted","Data":"c5328bac83b80ef56198cf5c7d4955df1eee6047efa0ef39220d6d158855cbd8"} Nov 03 22:47:52.645795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:52.645773 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" event={"ID":"a68a75c6-1469-4ce8-b300-e010e6b48b19","Type":"ContainerStarted","Data":"718a6e36be0aa121e7206ae867caec6d467e14cfe1ee6957cf16ba41c16a9d82"} Nov 03 22:47:52.690924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:52.690875 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" podStartSLOduration=1.690861592 podStartE2EDuration="1.690861592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:47:52.686710142 +0000 UTC m=+2099.090651117" watchObservedRunningTime="2026-11-03 22:47:52.690861592 +0000 UTC m=+2099.094802567" Nov 03 22:47:53.656752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:53.656717 4903 generic.go:356] "Generic (PLEG): container finished" podID="a68a75c6-1469-4ce8-b300-e010e6b48b19" containerID="c5328bac83b80ef56198cf5c7d4955df1eee6047efa0ef39220d6d158855cbd8" exitCode=0 Nov 03 22:47:53.657205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:53.656796 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" event={"ID":"a68a75c6-1469-4ce8-b300-e010e6b48b19","Type":"ContainerDied","Data":"c5328bac83b80ef56198cf5c7d4955df1eee6047efa0ef39220d6d158855cbd8"} Nov 03 22:47:53.810328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:53.806593 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" podUID="1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" containerName="main" containerID="cri-o://8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4" gracePeriod=45 Nov 03 22:47:54.135069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.135043 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:47:54.203186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.203107 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:47:54.206867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.206843 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:47:54.213255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.213232 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:47:54.218342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.218317 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:47:54.301488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.301456 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-home\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-home\") pod \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " Nov 03 22:47:54.301668 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.301578 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-dshm\") pod \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " Nov 03 22:47:54.301763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.301668 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tmp-dir\") pod \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " Nov 03 22:47:54.301763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.301749 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-kube-api-access-9bx8d\" (UniqueName: \"kubernetes.io/projected/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-kube-api-access-9bx8d\") pod \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " Nov 03 22:47:54.301897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.301769 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-home" pod "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" (UID: "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:54.301897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.301795 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-model-cache\") pod \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " Nov 03 22:47:54.302019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.301923 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tmp-dir" pod "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" (UID: "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:54.302019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.301936 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tls-certs\") pod \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\" (UID: \"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e\") " Nov 03 22:47:54.302174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.302141 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-model-cache" pod "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" (UID: "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:54.302586 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.302565 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:54.302677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.302587 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:54.302677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.302625 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:54.307919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.307895 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-dshm" pod "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" (UID: "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:47:54.308017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.307952 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-kube-api-access-9bx8d" pod "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" (UID: "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e"). InnerVolumeSpecName "kube-api-access-9bx8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:54.308017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.307980 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tls-certs" pod "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" (UID: "1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:54.404259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.404225 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:54.404259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.404258 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:54.404483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.404272 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bx8d\" (UniqueName: \"kubernetes.io/projected/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e-kube-api-access-9bx8d\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:54.468868 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.468767 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" podUID="8ac290e6-c72c-4184-9e06-e2e9af77eac2" containerName="payload-processing" containerID="cri-o://2886b5f17189d074f6c7b145741ff805d6005b5bd83575404620f76fec86323e" gracePeriod=25 Nov 03 22:47:54.510474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.510429 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" podUID="9cf0936b-4b43-4543-9787-8ef2de2ae70f" containerName="payload-pre-processing" containerID="cri-o://8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7" gracePeriod=25 Nov 03 22:47:54.675616 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.675569 4903 generic.go:356] "Generic (PLEG): container finished" podID="8ac290e6-c72c-4184-9e06-e2e9af77eac2" containerID="2886b5f17189d074f6c7b145741ff805d6005b5bd83575404620f76fec86323e" exitCode=0 Nov 03 22:47:54.676067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.675655 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" event={"ID":"8ac290e6-c72c-4184-9e06-e2e9af77eac2","Type":"ContainerDied","Data":"2886b5f17189d074f6c7b145741ff805d6005b5bd83575404620f76fec86323e"} Nov 03 22:47:54.677798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.677771 4903 generic.go:356] "Generic (PLEG): container finished" podID="1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" containerID="8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4" exitCode=0 Nov 03 22:47:54.677944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.677824 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" Nov 03 22:47:54.677944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.677843 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" event={"ID":"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e","Type":"ContainerDied","Data":"8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4"} Nov 03 22:47:54.677944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.677909 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4" event={"ID":"1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e","Type":"ContainerDied","Data":"2eb322032cb61af36a7bfed7c31c950ab235735216e8b1d4ea88b2cd73e58fad"} Nov 03 22:47:54.677944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.677934 4903 scope.go:122] "RemoveContainer" containerID="8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4" Nov 03 22:47:54.694508 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.693745 4903 scope.go:122] "RemoveContainer" containerID="8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4" Nov 03 22:47:54.694508 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:47:54.694087 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4\": container with ID starting with 8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4 not found: ID does not exist" containerID="8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4" Nov 03 22:47:54.694508 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.694128 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4"} err="failed to get container status \"8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4\": rpc error: code = NotFound desc = could not find container \"8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4\": container with ID starting with 8fb6cda1f241e88c3d6dd03ec7fe0d99cda26fb0369b98517592841fcd7c2dd4 not found: ID does not exist" Nov 03 22:47:54.736426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.736339 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4"] Nov 03 22:47:54.744604 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.744580 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0a7065/test-model-0a7065-kserve-5964946f49-k67f4"] Nov 03 22:47:54.766476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.764774 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:47:54.912072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.912042 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac290e6-c72c-4184-9e06-e2e9af77eac2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac290e6-c72c-4184-9e06-e2e9af77eac2-plugins-config-volume\") pod \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\" (UID: \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\") " Nov 03 22:47:54.912398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.912377 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac290e6-c72c-4184-9e06-e2e9af77eac2-kube-api-access-6xs5l\" (UniqueName: \"kubernetes.io/projected/8ac290e6-c72c-4184-9e06-e2e9af77eac2-kube-api-access-6xs5l\") pod \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\" (UID: \"8ac290e6-c72c-4184-9e06-e2e9af77eac2\") " Nov 03 22:47:54.912502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.912475 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac290e6-c72c-4184-9e06-e2e9af77eac2-plugins-config-volume" pod "8ac290e6-c72c-4184-9e06-e2e9af77eac2" (UID: "8ac290e6-c72c-4184-9e06-e2e9af77eac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:54.913078 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.913056 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac290e6-c72c-4184-9e06-e2e9af77eac2-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:54.914537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.914446 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac290e6-c72c-4184-9e06-e2e9af77eac2-kube-api-access-6xs5l" pod "8ac290e6-c72c-4184-9e06-e2e9af77eac2" (UID: "8ac290e6-c72c-4184-9e06-e2e9af77eac2"). InnerVolumeSpecName "kube-api-access-6xs5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:54.942669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:54.942647 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:55.015074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.014991 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xs5l\" (UniqueName: \"kubernetes.io/projected/8ac290e6-c72c-4184-9e06-e2e9af77eac2-kube-api-access-6xs5l\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:55.116237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.116198 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68a75c6-1469-4ce8-b300-e010e6b48b19-kube-api-access-5v5wq\" (UniqueName: \"kubernetes.io/projected/a68a75c6-1469-4ce8-b300-e010e6b48b19-kube-api-access-5v5wq\") pod \"a68a75c6-1469-4ce8-b300-e010e6b48b19\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " Nov 03 22:47:55.116470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.116257 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-xks-ca\") pod \"a68a75c6-1469-4ce8-b300-e010e6b48b19\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " Nov 03 22:47:55.116654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.116628 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-openshift-ca\") pod \"a68a75c6-1469-4ce8-b300-e010e6b48b19\" (UID: \"a68a75c6-1469-4ce8-b300-e010e6b48b19\") " Nov 03 22:47:55.117009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.116984 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-openshift-ca" pod "a68a75c6-1469-4ce8-b300-e010e6b48b19" (UID: "a68a75c6-1469-4ce8-b300-e010e6b48b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:47:55.117272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.117250 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:55.118546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.118518 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-xks-ca" pod "a68a75c6-1469-4ce8-b300-e010e6b48b19" (UID: "a68a75c6-1469-4ce8-b300-e010e6b48b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:47:55.118760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.118741 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68a75c6-1469-4ce8-b300-e010e6b48b19-kube-api-access-5v5wq" pod "a68a75c6-1469-4ce8-b300-e010e6b48b19" (UID: "a68a75c6-1469-4ce8-b300-e010e6b48b19"). InnerVolumeSpecName "kube-api-access-5v5wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:47:55.218373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.218343 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v5wq\" (UniqueName: \"kubernetes.io/projected/a68a75c6-1469-4ce8-b300-e010e6b48b19-kube-api-access-5v5wq\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:55.218373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.218370 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68a75c6-1469-4ce8-b300-e010e6b48b19-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:47:55.688986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.688957 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" Nov 03 22:47:55.688986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.688965 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59" event={"ID":"a68a75c6-1469-4ce8-b300-e010e6b48b19","Type":"ContainerDied","Data":"718a6e36be0aa121e7206ae867caec6d467e14cfe1ee6957cf16ba41c16a9d82"} Nov 03 22:47:55.689543 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.689018 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718a6e36be0aa121e7206ae867caec6d467e14cfe1ee6957cf16ba41c16a9d82" Nov 03 22:47:55.690437 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.690413 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" event={"ID":"8ac290e6-c72c-4184-9e06-e2e9af77eac2","Type":"ContainerDied","Data":"c624fc8bb038d384f391d67d44c72e984c0f36688f5dfa8df9b658f917257a16"} Nov 03 22:47:55.690563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.690467 4903 scope.go:122] "RemoveContainer" containerID="2886b5f17189d074f6c7b145741ff805d6005b5bd83575404620f76fec86323e" Nov 03 22:47:55.690563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.690424 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm" Nov 03 22:47:55.775915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.775879 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" path="/var/lib/kubelet/pods/1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e/volumes" Nov 03 22:47:55.778563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.778539 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm"] Nov 03 22:47:55.782864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:55.782843 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b1ec3a-5c67ddd778-kzpcm"] Nov 03 22:47:57.742313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:57.740463 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac290e6-c72c-4184-9e06-e2e9af77eac2" path="/var/lib/kubelet/pods/8ac290e6-c72c-4184-9e06-e2e9af77eac2/volumes" Nov 03 22:47:58.541398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:58.541368 4903 scope.go:122] "RemoveContainer" containerID="fe47d2caad43d4c3f6f691d072e8ebe607031bfc9a824216c99d8809fd26e2cc" Nov 03 22:47:59.008431 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:47:59.008389 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" podUID="9cf0936b-4b43-4543-9787-8ef2de2ae70f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:48:04.376973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:04.376936 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx"] Nov 03 22:48:04.386587 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:04.386557 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1ec3a-rc9cx"] Nov 03 22:48:05.738379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:05.738337 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a41773-99db-42c3-99c3-aa5f834f8682" path="/var/lib/kubelet/pods/49a41773-99db-42c3-99c3-aa5f834f8682/volumes" Nov 03 22:48:09.000204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:09.000156 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" podUID="9cf0936b-4b43-4543-9787-8ef2de2ae70f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:48:11.390578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.390543 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2"] Nov 03 22:48:11.391383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.391351 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" containerID="cri-o://6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9" gracePeriod=30 Nov 03 22:48:11.861491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.861455 4903 patch_prober.go:29] interesting pod/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:48:11.861686 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.861528 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:48:11.965893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.965859 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh"] Nov 03 22:48:11.966755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.966738 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a68a75c6-1469-4ce8-b300-e010e6b48b19" containerName="revoke-keys" Nov 03 22:48:11.966805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.966756 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68a75c6-1469-4ce8-b300-e010e6b48b19" containerName="revoke-keys" Nov 03 22:48:11.966805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.966779 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8ac290e6-c72c-4184-9e06-e2e9af77eac2" containerName="payload-processing" Nov 03 22:48:11.966805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.966785 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac290e6-c72c-4184-9e06-e2e9af77eac2" containerName="payload-processing" Nov 03 22:48:11.966805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.966792 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" containerName="main" Nov 03 22:48:11.966805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.966796 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" containerName="main" Nov 03 22:48:11.966988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.966948 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68a75c6-1469-4ce8-b300-e010e6b48b19" containerName="revoke-keys" Nov 03 22:48:11.966988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.966961 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac290e6-c72c-4184-9e06-e2e9af77eac2" containerName="payload-processing" Nov 03 22:48:11.966988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.966966 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea7032d-33e2-4ea1-ae81-edf2f54d6c0e" containerName="main" Nov 03 22:48:11.972945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.972920 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:11.989034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.989004 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:48:11.989183 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.989141 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:48:11.995252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:11.995225 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh"] Nov 03 22:48:12.071534 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.071501 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:48:12.071722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.071543 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:48:12.086439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.086408 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhp4\" (UniqueName: \"kubernetes.io/projected/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-kube-api-access-dwhp4\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.086600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.086455 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.086649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.086588 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.189018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.188971 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.194650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.194613 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhp4\" (UniqueName: \"kubernetes.io/projected/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-kube-api-access-dwhp4\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.194856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.194687 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.195206 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.190058 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.198379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.198353 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.220000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.219957 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhp4\" (UniqueName: \"kubernetes.io/projected/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-kube-api-access-dwhp4\") pod \"maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.287883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.287848 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:12.571487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.571379 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9"] Nov 03 22:48:12.578081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.578055 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.600728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.600662 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9"] Nov 03 22:48:12.620389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.616050 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh"] Nov 03 22:48:12.703557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.703520 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fca835c-ggmt9\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.703805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.703784 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fca835c-ggmt9\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.703928 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.703908 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9hwz\" (UniqueName: \"kubernetes.io/projected/125cfca1-f3d2-4ca7-8c21-f6509297e54f-kube-api-access-p9hwz\") pod \"maas-api-revoke-sub-0fca835c-ggmt9\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.805400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.805364 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fca835c-ggmt9\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.805599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.805482 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9hwz\" (UniqueName: \"kubernetes.io/projected/125cfca1-f3d2-4ca7-8c21-f6509297e54f-kube-api-access-p9hwz\") pod \"maas-api-revoke-sub-0fca835c-ggmt9\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.805655 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.805602 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fca835c-ggmt9\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.806139 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.806119 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fca835c-ggmt9\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.808022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.808000 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fca835c-ggmt9\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.827743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.827683 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9hwz\" (UniqueName: \"kubernetes.io/projected/125cfca1-f3d2-4ca7-8c21-f6509297e54f-kube-api-access-p9hwz\") pod \"maas-api-revoke-sub-0fca835c-ggmt9\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:12.920579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:12.920547 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:13.000691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:13.000658 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" event={"ID":"2dd29f53-54d6-4e4d-93d0-ea580c7da96a","Type":"ContainerStarted","Data":"5afe8794d71f75d8dc537eead606b7e728ce201423907324c79b52d37262993e"} Nov 03 22:48:13.294852 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:48:13.294819 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125cfca1_f3d2_4ca7_8c21_f6509297e54f.slice/crio-c0cdb30522c2135902937e8cc43e949045d6ba1357ea96d502a863d7057d523b WatchSource:0}: Error finding container c0cdb30522c2135902937e8cc43e949045d6ba1357ea96d502a863d7057d523b: Status 404 returned error can't find the container with id c0cdb30522c2135902937e8cc43e949045d6ba1357ea96d502a863d7057d523b Nov 03 22:48:13.297031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:13.297006 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9"] Nov 03 22:48:14.011240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.011204 4903 generic.go:356] "Generic (PLEG): container finished" podID="2dd29f53-54d6-4e4d-93d0-ea580c7da96a" containerID="df8cf465b5b05eb75ab06ea9014cdb3690bdd3ff2139251441ff474733022473" exitCode=0 Nov 03 22:48:14.011738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.011273 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" event={"ID":"2dd29f53-54d6-4e4d-93d0-ea580c7da96a","Type":"ContainerDied","Data":"df8cf465b5b05eb75ab06ea9014cdb3690bdd3ff2139251441ff474733022473"} Nov 03 22:48:14.013349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.013324 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" event={"ID":"125cfca1-f3d2-4ca7-8c21-f6509297e54f","Type":"ContainerStarted","Data":"85c88b9ad10254270846a523b142b7b877a8b6020cb1afa87d103ad77b50dc00"} Nov 03 22:48:14.013474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.013366 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" event={"ID":"125cfca1-f3d2-4ca7-8c21-f6509297e54f","Type":"ContainerStarted","Data":"c0cdb30522c2135902937e8cc43e949045d6ba1357ea96d502a863d7057d523b"} Nov 03 22:48:14.085621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.085569 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" podStartSLOduration=2.085552147 podStartE2EDuration="2.085552147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:48:14.083539923 +0000 UTC m=+2120.487480897" watchObservedRunningTime="2026-11-03 22:48:14.085552147 +0000 UTC m=+2120.489493123" Nov 03 22:48:14.830000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.829967 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:48:14.942824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.942797 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf0936b-4b43-4543-9787-8ef2de2ae70f-kube-api-access-hjjdt\" (UniqueName: \"kubernetes.io/projected/9cf0936b-4b43-4543-9787-8ef2de2ae70f-kube-api-access-hjjdt\") pod \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\" (UID: \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\") " Nov 03 22:48:14.943017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.942937 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf0936b-4b43-4543-9787-8ef2de2ae70f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf0936b-4b43-4543-9787-8ef2de2ae70f-plugins-config-volume\") pod \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\" (UID: \"9cf0936b-4b43-4543-9787-8ef2de2ae70f\") " Nov 03 22:48:14.943344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.943315 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf0936b-4b43-4543-9787-8ef2de2ae70f-plugins-config-volume" pod "9cf0936b-4b43-4543-9787-8ef2de2ae70f" (UID: "9cf0936b-4b43-4543-9787-8ef2de2ae70f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:14.943679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.943663 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf0936b-4b43-4543-9787-8ef2de2ae70f-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:14.944960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:14.944941 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf0936b-4b43-4543-9787-8ef2de2ae70f-kube-api-access-hjjdt" pod "9cf0936b-4b43-4543-9787-8ef2de2ae70f" (UID: "9cf0936b-4b43-4543-9787-8ef2de2ae70f"). InnerVolumeSpecName "kube-api-access-hjjdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:15.023343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.023305 4903 generic.go:356] "Generic (PLEG): container finished" podID="125cfca1-f3d2-4ca7-8c21-f6509297e54f" containerID="85c88b9ad10254270846a523b142b7b877a8b6020cb1afa87d103ad77b50dc00" exitCode=0 Nov 03 22:48:15.023791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.023398 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" event={"ID":"125cfca1-f3d2-4ca7-8c21-f6509297e54f","Type":"ContainerDied","Data":"85c88b9ad10254270846a523b142b7b877a8b6020cb1afa87d103ad77b50dc00"} Nov 03 22:48:15.024823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.024799 4903 generic.go:356] "Generic (PLEG): container finished" podID="9cf0936b-4b43-4543-9787-8ef2de2ae70f" containerID="8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7" exitCode=0 Nov 03 22:48:15.024971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.024844 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" Nov 03 22:48:15.024971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.024877 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" event={"ID":"9cf0936b-4b43-4543-9787-8ef2de2ae70f","Type":"ContainerDied","Data":"8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7"} Nov 03 22:48:15.024971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.024920 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx" event={"ID":"9cf0936b-4b43-4543-9787-8ef2de2ae70f","Type":"ContainerDied","Data":"a38443fdc585820dcfdff527dab718b6c85fdb01c1d785679781ac0c4aeba425"} Nov 03 22:48:15.024971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.024943 4903 scope.go:122] "RemoveContainer" containerID="8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7" Nov 03 22:48:15.036453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.036437 4903 scope.go:122] "RemoveContainer" containerID="8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7" Nov 03 22:48:15.036737 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:48:15.036719 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7\": container with ID starting with 8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7 not found: ID does not exist" containerID="8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7" Nov 03 22:48:15.036844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.036749 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7"} err="failed to get container status \"8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7\": rpc error: code = NotFound desc = could not find container \"8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7\": container with ID starting with 8d3a49b228446810e029e05c24b5180828d92954a51e6b7a348c767a770b8ce7 not found: ID does not exist" Nov 03 22:48:15.045184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.045158 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjjdt\" (UniqueName: \"kubernetes.io/projected/9cf0936b-4b43-4543-9787-8ef2de2ae70f-kube-api-access-hjjdt\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:15.113155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.102567 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx"] Nov 03 22:48:15.123501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.118718 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b1ec3a-6fdb57959f-fxmdx"] Nov 03 22:48:15.187831 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.187803 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:15.257620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.257511 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-openshift-ca\") pod \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " Nov 03 22:48:15.257791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.257675 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-xks-ca\") pod \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " Nov 03 22:48:15.257791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.257733 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-kube-api-access-dwhp4\" (UniqueName: \"kubernetes.io/projected/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-kube-api-access-dwhp4\") pod \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\" (UID: \"2dd29f53-54d6-4e4d-93d0-ea580c7da96a\") " Nov 03 22:48:15.258056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.258026 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-openshift-ca" pod "2dd29f53-54d6-4e4d-93d0-ea580c7da96a" (UID: "2dd29f53-54d6-4e4d-93d0-ea580c7da96a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:15.258785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.258758 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:15.260056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.260027 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-xks-ca" pod "2dd29f53-54d6-4e4d-93d0-ea580c7da96a" (UID: "2dd29f53-54d6-4e4d-93d0-ea580c7da96a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:15.260056 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.260046 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-kube-api-access-dwhp4" pod "2dd29f53-54d6-4e4d-93d0-ea580c7da96a" (UID: "2dd29f53-54d6-4e4d-93d0-ea580c7da96a"). InnerVolumeSpecName "kube-api-access-dwhp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:15.360546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.360480 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:15.360546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.360515 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhp4\" (UniqueName: \"kubernetes.io/projected/2dd29f53-54d6-4e4d-93d0-ea580c7da96a-kube-api-access-dwhp4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:15.738404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:15.738367 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf0936b-4b43-4543-9787-8ef2de2ae70f" path="/var/lib/kubelet/pods/9cf0936b-4b43-4543-9787-8ef2de2ae70f/volumes" Nov 03 22:48:16.039599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.039517 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" Nov 03 22:48:16.039599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.039539 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh" event={"ID":"2dd29f53-54d6-4e4d-93d0-ea580c7da96a","Type":"ContainerDied","Data":"5afe8794d71f75d8dc537eead606b7e728ce201423907324c79b52d37262993e"} Nov 03 22:48:16.040101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.039602 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5afe8794d71f75d8dc537eead606b7e728ce201423907324c79b52d37262993e" Nov 03 22:48:16.216428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.216400 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:16.376894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.376815 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-openshift-ca\") pod \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " Nov 03 22:48:16.377060 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.376972 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-xks-ca\") pod \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " Nov 03 22:48:16.377060 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.377038 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125cfca1-f3d2-4ca7-8c21-f6509297e54f-kube-api-access-p9hwz\" (UniqueName: \"kubernetes.io/projected/125cfca1-f3d2-4ca7-8c21-f6509297e54f-kube-api-access-p9hwz\") pod \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\" (UID: \"125cfca1-f3d2-4ca7-8c21-f6509297e54f\") " Nov 03 22:48:16.377256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.377228 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-openshift-ca" pod "125cfca1-f3d2-4ca7-8c21-f6509297e54f" (UID: "125cfca1-f3d2-4ca7-8c21-f6509297e54f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:16.377656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.377626 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.379128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.379102 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-xks-ca" pod "125cfca1-f3d2-4ca7-8c21-f6509297e54f" (UID: "125cfca1-f3d2-4ca7-8c21-f6509297e54f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:16.379375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.379356 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125cfca1-f3d2-4ca7-8c21-f6509297e54f-kube-api-access-p9hwz" pod "125cfca1-f3d2-4ca7-8c21-f6509297e54f" (UID: "125cfca1-f3d2-4ca7-8c21-f6509297e54f"). InnerVolumeSpecName "kube-api-access-p9hwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:16.479512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.479480 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/125cfca1-f3d2-4ca7-8c21-f6509297e54f-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.479512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.479507 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9hwz\" (UniqueName: \"kubernetes.io/projected/125cfca1-f3d2-4ca7-8c21-f6509297e54f-kube-api-access-p9hwz\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.719581 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.719557 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:48:16.887343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.886451 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-token\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-token\") pod \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " Nov 03 22:48:16.887343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.886679 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-data\") pod \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " Nov 03 22:48:16.887343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.886842 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-envoy\") pod \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " Nov 03 22:48:16.887343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.886948 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-certs\") pod \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " Nov 03 22:48:16.887343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.886995 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-kube-api-access-pxtlp\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-kube-api-access-pxtlp\") pod \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " Nov 03 22:48:16.887343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.887056 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-socket\") pod \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " Nov 03 22:48:16.887343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.887213 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-podinfo\") pod \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " Nov 03 22:48:16.887917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.887359 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istiod-ca-cert\") pod \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " Nov 03 22:48:16.887917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.887535 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-credential-socket\") pod \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\" (UID: \"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e\") " Nov 03 22:48:16.892343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.888641 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istiod-ca-cert" pod "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" (UID: "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:16.892343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.888967 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-credential-socket" pod "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" (UID: "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:16.892343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.889153 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-certs" pod "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" (UID: "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:16.892343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.889363 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-socket" pod "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" (UID: "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:16.892343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.890608 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.892343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.890632 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.892343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.890647 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.892343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.890661 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.892343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.892004 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-data" pod "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" (UID: "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:16.923386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.916144 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-podinfo" pod "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" (UID: "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:48:16.923386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.916324 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-envoy" pod "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" (UID: "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:48:16.923386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.916482 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-token" pod "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" (UID: "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:16.953381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.948639 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-kube-api-access-pxtlp" pod "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" (UID: "6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e"). InnerVolumeSpecName "kube-api-access-pxtlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:16.992781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.992749 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.992781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.992774 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.992781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.992786 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.993113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.992800 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:16.993113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:16.992810 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxtlp\" (UniqueName: \"kubernetes.io/projected/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e-kube-api-access-pxtlp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:17.053382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.053340 4903 generic.go:356] "Generic (PLEG): container finished" podID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerID="6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9" exitCode=0 Nov 03 22:48:17.053863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.053397 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" Nov 03 22:48:17.053863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.053420 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" event={"ID":"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e","Type":"ContainerDied","Data":"6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9"} Nov 03 22:48:17.053863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.053477 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2" event={"ID":"6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e","Type":"ContainerDied","Data":"c75a8e78a7477215547f14c913b2e0d4e6b0048b232b62455caf4c622705b7c3"} Nov 03 22:48:17.053863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.053500 4903 scope.go:122] "RemoveContainer" containerID="6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9" Nov 03 22:48:17.055219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.055204 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" Nov 03 22:48:17.055350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.055265 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9" event={"ID":"125cfca1-f3d2-4ca7-8c21-f6509297e54f","Type":"ContainerDied","Data":"c0cdb30522c2135902937e8cc43e949045d6ba1357ea96d502a863d7057d523b"} Nov 03 22:48:17.055350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.055320 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0cdb30522c2135902937e8cc43e949045d6ba1357ea96d502a863d7057d523b" Nov 03 22:48:17.066884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.066867 4903 scope.go:122] "RemoveContainer" containerID="349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841" Nov 03 22:48:17.077505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.077485 4903 scope.go:122] "RemoveContainer" containerID="6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9" Nov 03 22:48:17.077764 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:48:17.077743 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9\": container with ID starting with 6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9 not found: ID does not exist" containerID="6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9" Nov 03 22:48:17.077837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.077773 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9"} err="failed to get container status \"6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9\": rpc error: code = NotFound desc = could not find container \"6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9\": container with ID starting with 6cf163b2363d74992f5dcea83f61c85b8cc33f383dad47daaf6edea1f266f4c9 not found: ID does not exist" Nov 03 22:48:17.077837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.077791 4903 scope.go:122] "RemoveContainer" containerID="349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841" Nov 03 22:48:17.078031 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:48:17.078013 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841\": container with ID starting with 349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841 not found: ID does not exist" containerID="349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841" Nov 03 22:48:17.078090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.078040 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841"} err="failed to get container status \"349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841\": rpc error: code = NotFound desc = could not find container \"349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841\": container with ID starting with 349dae1d8750ca72bc09d0eb4725b5d1249049c5eeda9728700d2e44a98ad841 not found: ID does not exist" Nov 03 22:48:17.118099 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.118064 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2"] Nov 03 22:48:17.129755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.129725 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b1ec3a-openshift-default-7675569d9-qjkx2"] Nov 03 22:48:17.740132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:17.740096 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" path="/var/lib/kubelet/pods/6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e/volumes" Nov 03 22:48:18.386336 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:48:18.386269 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:48:22.403036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:22.402999 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k"] Nov 03 22:48:25.205434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.205397 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx"] Nov 03 22:48:25.206474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206451 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2dd29f53-54d6-4e4d-93d0-ea580c7da96a" containerName="revoke-keys" Nov 03 22:48:25.206474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206473 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd29f53-54d6-4e4d-93d0-ea580c7da96a" containerName="revoke-keys" Nov 03 22:48:25.206658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206513 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" Nov 03 22:48:25.206658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206521 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" Nov 03 22:48:25.206658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206583 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" Nov 03 22:48:25.206658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206591 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" Nov 03 22:48:25.206658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206601 4903 cpu_manager.go:414] "containerMap: removing container" podUID="125cfca1-f3d2-4ca7-8c21-f6509297e54f" containerName="revoke-keys" Nov 03 22:48:25.206658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206608 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125cfca1-f3d2-4ca7-8c21-f6509297e54f" containerName="revoke-keys" Nov 03 22:48:25.206658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206624 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9cf0936b-4b43-4543-9787-8ef2de2ae70f" containerName="payload-pre-processing" Nov 03 22:48:25.206658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206632 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf0936b-4b43-4543-9787-8ef2de2ae70f" containerName="payload-pre-processing" Nov 03 22:48:25.207066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206849 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" Nov 03 22:48:25.207066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206862 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="125cfca1-f3d2-4ca7-8c21-f6509297e54f" containerName="revoke-keys" Nov 03 22:48:25.207066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206875 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf0936b-4b43-4543-9787-8ef2de2ae70f" containerName="payload-pre-processing" Nov 03 22:48:25.207066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206888 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd29f53-54d6-4e4d-93d0-ea580c7da96a" containerName="revoke-keys" Nov 03 22:48:25.207066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.206901 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6694caf3-1707-4ae5-a4b8-d9bc3e16ce8e" containerName="istio-proxy" Nov 03 22:48:25.211168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.211149 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.227910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.227880 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx"] Nov 03 22:48:25.232108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.232081 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:48:25.232108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.232090 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:48:25.232328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.232223 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:48:25.232328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.232223 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:48:25.270357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.270289 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.270536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.270465 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpz5v\" (UniqueName: \"kubernetes.io/projected/3b92bfc6-9b8f-4836-ae5c-325c53886b49-kube-api-access-bpz5v\") pod \"maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.270536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.270510 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.372393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.372360 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.372584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.372505 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpz5v\" (UniqueName: \"kubernetes.io/projected/3b92bfc6-9b8f-4836-ae5c-325c53886b49-kube-api-access-bpz5v\") pod \"maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.372584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.372549 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.373195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.373172 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.375190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.375171 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.393684 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.393662 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpz5v\" (UniqueName: \"kubernetes.io/projected/3b92bfc6-9b8f-4836-ae5c-325c53886b49-kube-api-access-bpz5v\") pod \"maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.526687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.526603 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:25.844364 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:48:25.844334 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b92bfc6_9b8f_4836_ae5c_325c53886b49.slice/crio-d4e4d7c7e9cfe373ca51c6deaf8252897fc563fd6c878b2f19e17ffcb06bcd7c WatchSource:0}: Error finding container d4e4d7c7e9cfe373ca51c6deaf8252897fc563fd6c878b2f19e17ffcb06bcd7c: Status 404 returned error can't find the container with id d4e4d7c7e9cfe373ca51c6deaf8252897fc563fd6c878b2f19e17ffcb06bcd7c Nov 03 22:48:25.846011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:25.845984 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx"] Nov 03 22:48:26.158622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:26.158586 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" event={"ID":"3b92bfc6-9b8f-4836-ae5c-325c53886b49","Type":"ContainerStarted","Data":"d4e4d7c7e9cfe373ca51c6deaf8252897fc563fd6c878b2f19e17ffcb06bcd7c"} Nov 03 22:48:27.168181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.168144 4903 generic.go:356] "Generic (PLEG): container finished" podID="3b92bfc6-9b8f-4836-ae5c-325c53886b49" containerID="85db199ca35d0d658df4d84eba4482fa32a0c917b1b574facc5b3cfa207b3f4f" exitCode=0 Nov 03 22:48:27.168602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.168204 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" event={"ID":"3b92bfc6-9b8f-4836-ae5c-325c53886b49","Type":"ContainerDied","Data":"85db199ca35d0d658df4d84eba4482fa32a0c917b1b574facc5b3cfa207b3f4f"} Nov 03 22:48:27.403734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.403685 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" podUID="642bd136-2764-4607-9048-d9103f23d8a6" containerName="payload-processing" containerID="cri-o://ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438" gracePeriod=25 Nov 03 22:48:27.695025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.694956 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:48:27.813734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.813704 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/642bd136-2764-4607-9048-d9103f23d8a6-kube-api-access-9528b\" (UniqueName: \"kubernetes.io/projected/642bd136-2764-4607-9048-d9103f23d8a6-kube-api-access-9528b\") pod \"642bd136-2764-4607-9048-d9103f23d8a6\" (UID: \"642bd136-2764-4607-9048-d9103f23d8a6\") " Nov 03 22:48:27.813912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.813829 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/642bd136-2764-4607-9048-d9103f23d8a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642bd136-2764-4607-9048-d9103f23d8a6-plugins-config-volume\") pod \"642bd136-2764-4607-9048-d9103f23d8a6\" (UID: \"642bd136-2764-4607-9048-d9103f23d8a6\") " Nov 03 22:48:27.814209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.814184 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/642bd136-2764-4607-9048-d9103f23d8a6-plugins-config-volume" pod "642bd136-2764-4607-9048-d9103f23d8a6" (UID: "642bd136-2764-4607-9048-d9103f23d8a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:27.814575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.814551 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/642bd136-2764-4607-9048-d9103f23d8a6-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:27.815890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.815866 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/642bd136-2764-4607-9048-d9103f23d8a6-kube-api-access-9528b" pod "642bd136-2764-4607-9048-d9103f23d8a6" (UID: "642bd136-2764-4607-9048-d9103f23d8a6"). InnerVolumeSpecName "kube-api-access-9528b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:27.916220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:27.916185 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9528b\" (UniqueName: \"kubernetes.io/projected/642bd136-2764-4607-9048-d9103f23d8a6-kube-api-access-9528b\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:28.178328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.178278 4903 generic.go:356] "Generic (PLEG): container finished" podID="642bd136-2764-4607-9048-d9103f23d8a6" containerID="ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438" exitCode=0 Nov 03 22:48:28.178852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.178538 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" event={"ID":"642bd136-2764-4607-9048-d9103f23d8a6","Type":"ContainerDied","Data":"ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438"} Nov 03 22:48:28.178852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.178571 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" Nov 03 22:48:28.178852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.178630 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k" event={"ID":"642bd136-2764-4607-9048-d9103f23d8a6","Type":"ContainerDied","Data":"c3d681d1a8c64fadf9fa526f13b862d5627219ee0dd8cd596e234b31138f7ca5"} Nov 03 22:48:28.178852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.178663 4903 scope.go:122] "RemoveContainer" containerID="ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438" Nov 03 22:48:28.194886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.194864 4903 scope.go:122] "RemoveContainer" containerID="ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438" Nov 03 22:48:28.195216 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:48:28.195192 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438\": container with ID starting with ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438 not found: ID does not exist" containerID="ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438" Nov 03 22:48:28.195356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.195222 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438"} err="failed to get container status \"ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438\": rpc error: code = NotFound desc = could not find container \"ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438\": container with ID starting with ed9a7994f2a0215fabb9860d68e97120ec92f0aded27fa5d69bde3b66ae65438 not found: ID does not exist" Nov 03 22:48:28.299265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.299188 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k"] Nov 03 22:48:28.320903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.320865 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a7065-694768f4b4-5fq8k"] Nov 03 22:48:28.389551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.389522 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:28.525122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.525027 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-xks-ca\") pod \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " Nov 03 22:48:28.525122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.525099 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b92bfc6-9b8f-4836-ae5c-325c53886b49-kube-api-access-bpz5v\" (UniqueName: \"kubernetes.io/projected/3b92bfc6-9b8f-4836-ae5c-325c53886b49-kube-api-access-bpz5v\") pod \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " Nov 03 22:48:28.525412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.525237 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-openshift-ca\") pod \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\" (UID: \"3b92bfc6-9b8f-4836-ae5c-325c53886b49\") " Nov 03 22:48:28.525735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.525709 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-openshift-ca" pod "3b92bfc6-9b8f-4836-ae5c-325c53886b49" (UID: "3b92bfc6-9b8f-4836-ae5c-325c53886b49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:48:28.526148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.526122 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:28.527276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.527252 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-xks-ca" pod "3b92bfc6-9b8f-4836-ae5c-325c53886b49" (UID: "3b92bfc6-9b8f-4836-ae5c-325c53886b49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:48:28.527466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.527442 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b92bfc6-9b8f-4836-ae5c-325c53886b49-kube-api-access-bpz5v" pod "3b92bfc6-9b8f-4836-ae5c-325c53886b49" (UID: "3b92bfc6-9b8f-4836-ae5c-325c53886b49"). InnerVolumeSpecName "kube-api-access-bpz5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:48:28.627607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.627572 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b92bfc6-9b8f-4836-ae5c-325c53886b49-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:28.627607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:28.627603 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpz5v\" (UniqueName: \"kubernetes.io/projected/3b92bfc6-9b8f-4836-ae5c-325c53886b49-kube-api-access-bpz5v\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:48:29.192355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:29.192321 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" Nov 03 22:48:29.192886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:29.192288 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx" event={"ID":"3b92bfc6-9b8f-4836-ae5c-325c53886b49","Type":"ContainerDied","Data":"d4e4d7c7e9cfe373ca51c6deaf8252897fc563fd6c878b2f19e17ffcb06bcd7c"} Nov 03 22:48:29.192886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:29.192485 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e4d7c7e9cfe373ca51c6deaf8252897fc563fd6c878b2f19e17ffcb06bcd7c" Nov 03 22:48:29.737396 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:29.737356 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="642bd136-2764-4607-9048-d9103f23d8a6" path="/var/lib/kubelet/pods/642bd136-2764-4607-9048-d9103f23d8a6/volumes" Nov 03 22:48:32.518141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:32.518100 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:48:35.664960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:35.664920 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b"] Nov 03 22:48:40.665617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:40.665571 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" podUID="f55e5d3f-2802-49a3-a784-602e3d1c1327" containerName="payload-pre-processing" containerID="cri-o://a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1" gracePeriod=25 Nov 03 22:48:45.745117 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:45.745080 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" podUID="f55e5d3f-2802-49a3-a784-602e3d1c1327" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:48:48.385484 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:48:48.385449 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:48:55.315979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:55.315945 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh"] Nov 03 22:48:55.325519 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:55.325480 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0a7065-nfbmh"] Nov 03 22:48:55.736391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:55.736358 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd29f53-54d6-4e4d-93d0-ea580c7da96a" path="/var/lib/kubelet/pods/2dd29f53-54d6-4e4d-93d0-ea580c7da96a/volumes" Nov 03 22:48:55.746150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:48:55.746109 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" podUID="f55e5d3f-2802-49a3-a784-602e3d1c1327" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:49:00.963986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:00.963958 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:49:01.060024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.059994 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55e5d3f-2802-49a3-a784-602e3d1c1327-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55e5d3f-2802-49a3-a784-602e3d1c1327-plugins-config-volume\") pod \"f55e5d3f-2802-49a3-a784-602e3d1c1327\" (UID: \"f55e5d3f-2802-49a3-a784-602e3d1c1327\") " Nov 03 22:49:01.060241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.060221 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55e5d3f-2802-49a3-a784-602e3d1c1327-kube-api-access-7rrk4\" (UniqueName: \"kubernetes.io/projected/f55e5d3f-2802-49a3-a784-602e3d1c1327-kube-api-access-7rrk4\") pod \"f55e5d3f-2802-49a3-a784-602e3d1c1327\" (UID: \"f55e5d3f-2802-49a3-a784-602e3d1c1327\") " Nov 03 22:49:01.060466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.060437 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55e5d3f-2802-49a3-a784-602e3d1c1327-plugins-config-volume" pod "f55e5d3f-2802-49a3-a784-602e3d1c1327" (UID: "f55e5d3f-2802-49a3-a784-602e3d1c1327"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:01.061106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.061079 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55e5d3f-2802-49a3-a784-602e3d1c1327-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:01.062739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.062713 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55e5d3f-2802-49a3-a784-602e3d1c1327-kube-api-access-7rrk4" pod "f55e5d3f-2802-49a3-a784-602e3d1c1327" (UID: "f55e5d3f-2802-49a3-a784-602e3d1c1327"). InnerVolumeSpecName "kube-api-access-7rrk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:01.163046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.163012 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rrk4\" (UniqueName: \"kubernetes.io/projected/f55e5d3f-2802-49a3-a784-602e3d1c1327-kube-api-access-7rrk4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:01.585473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.585438 4903 generic.go:356] "Generic (PLEG): container finished" podID="f55e5d3f-2802-49a3-a784-602e3d1c1327" containerID="a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1" exitCode=0 Nov 03 22:49:01.585665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.585489 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" Nov 03 22:49:01.585665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.585518 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" event={"ID":"f55e5d3f-2802-49a3-a784-602e3d1c1327","Type":"ContainerDied","Data":"a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1"} Nov 03 22:49:01.585665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.585603 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b" event={"ID":"f55e5d3f-2802-49a3-a784-602e3d1c1327","Type":"ContainerDied","Data":"ee98209422826f9d0986b591abaea2d257f1797818b9387c79687f827d00aafd"} Nov 03 22:49:01.585665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.585633 4903 scope.go:122] "RemoveContainer" containerID="a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1" Nov 03 22:49:01.598355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.598336 4903 scope.go:122] "RemoveContainer" containerID="a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1" Nov 03 22:49:01.598703 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:49:01.598683 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1\": container with ID starting with a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1 not found: ID does not exist" containerID="a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1" Nov 03 22:49:01.598760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.598711 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1"} err="failed to get container status \"a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1\": rpc error: code = NotFound desc = could not find container \"a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1\": container with ID starting with a073b7dde0f3896f8dda03ff7bd0037728c0a2d92952f8fc932ff3a2abb3dda1 not found: ID does not exist" Nov 03 22:49:01.634744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.634711 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b"] Nov 03 22:49:01.644830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.644801 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9e540d-67db6b6b55-f687b"] Nov 03 22:49:01.739383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:01.739347 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55e5d3f-2802-49a3-a784-602e3d1c1327" path="/var/lib/kubelet/pods/f55e5d3f-2802-49a3-a784-602e3d1c1327/volumes" Nov 03 22:49:02.694546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:02.694514 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx"] Nov 03 22:49:02.704512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:02.704482 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e540d-jvnzx"] Nov 03 22:49:03.740739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:03.740703 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b92bfc6-9b8f-4836-ae5c-325c53886b49" path="/var/lib/kubelet/pods/3b92bfc6-9b8f-4836-ae5c-325c53886b49/volumes" Nov 03 22:49:10.088731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:10.088693 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q"] Nov 03 22:49:10.089225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:10.088965 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" containerID="cri-o://035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4" gracePeriod=30 Nov 03 22:49:11.112817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.112782 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp"] Nov 03 22:49:11.113728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.113696 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f55e5d3f-2802-49a3-a784-602e3d1c1327" containerName="payload-pre-processing" Nov 03 22:49:11.113728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.113717 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55e5d3f-2802-49a3-a784-602e3d1c1327" containerName="payload-pre-processing" Nov 03 22:49:11.113946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.113798 4903 cpu_manager.go:414] "containerMap: removing container" podUID="642bd136-2764-4607-9048-d9103f23d8a6" containerName="payload-processing" Nov 03 22:49:11.113946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.113806 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="642bd136-2764-4607-9048-d9103f23d8a6" containerName="payload-processing" Nov 03 22:49:11.113946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.113835 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3b92bfc6-9b8f-4836-ae5c-325c53886b49" containerName="revoke-keys" Nov 03 22:49:11.113946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.113842 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b92bfc6-9b8f-4836-ae5c-325c53886b49" containerName="revoke-keys" Nov 03 22:49:11.114158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.114034 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b92bfc6-9b8f-4836-ae5c-325c53886b49" containerName="revoke-keys" Nov 03 22:49:11.114158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.114051 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55e5d3f-2802-49a3-a784-602e3d1c1327" containerName="payload-pre-processing" Nov 03 22:49:11.114158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.114075 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="642bd136-2764-4607-9048-d9103f23d8a6" containerName="payload-processing" Nov 03 22:49:11.119670 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.119648 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.139167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.139136 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp"] Nov 03 22:49:11.176566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.176533 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.176740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.176593 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.176740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.176709 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr256\" (UniqueName: \"kubernetes.io/projected/702af3e1-839e-4260-b7a1-3f335a313e5b-kube-api-access-cr256\") pod \"maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.201281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.201245 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:49:11.201680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.201608 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:49:11.210250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.210220 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:49:11.210573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.210553 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:49:11.354959 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.354912 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.355260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.355226 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.355447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.355409 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr256\" (UniqueName: \"kubernetes.io/projected/702af3e1-839e-4260-b7a1-3f335a313e5b-kube-api-access-cr256\") pod \"maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.356878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.356819 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.381125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.378372 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr256\" (UniqueName: \"kubernetes.io/projected/702af3e1-839e-4260-b7a1-3f335a313e5b-kube-api-access-cr256\") pod \"maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.381125 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.380014 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.443484 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.443453 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:11.767144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:11.767105 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp"] Nov 03 22:49:11.767842 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:49:11.767813 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702af3e1_839e_4260_b7a1_3f335a313e5b.slice/crio-dd53d69195041fb07672cf762d2b3d31da231dffe8b4afdc0177bbd4746f5c50 WatchSource:0}: Error finding container dd53d69195041fb07672cf762d2b3d31da231dffe8b4afdc0177bbd4746f5c50: Status 404 returned error can't find the container with id dd53d69195041fb07672cf762d2b3d31da231dffe8b4afdc0177bbd4746f5c50 Nov 03 22:49:12.734359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:12.734284 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" event={"ID":"702af3e1-839e-4260-b7a1-3f335a313e5b","Type":"ContainerStarted","Data":"1fba55516d01473198f5014ed853a9d4f0fa799a207c2a86ab2548b473236c02"} Nov 03 22:49:12.734359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:12.734358 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" event={"ID":"702af3e1-839e-4260-b7a1-3f335a313e5b","Type":"ContainerStarted","Data":"dd53d69195041fb07672cf762d2b3d31da231dffe8b4afdc0177bbd4746f5c50"} Nov 03 22:49:12.771966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:12.771911 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" podStartSLOduration=1.77189353 podStartE2EDuration="1.77189353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:49:12.771498604 +0000 UTC m=+2179.175439579" watchObservedRunningTime="2026-11-03 22:49:12.77189353 +0000 UTC m=+2179.175834505" Nov 03 22:49:13.744394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:13.744348 4903 generic.go:356] "Generic (PLEG): container finished" podID="702af3e1-839e-4260-b7a1-3f335a313e5b" containerID="1fba55516d01473198f5014ed853a9d4f0fa799a207c2a86ab2548b473236c02" exitCode=0 Nov 03 22:49:13.744856 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:13.744424 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" event={"ID":"702af3e1-839e-4260-b7a1-3f335a313e5b","Type":"ContainerDied","Data":"1fba55516d01473198f5014ed853a9d4f0fa799a207c2a86ab2548b473236c02"} Nov 03 22:49:14.961539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:14.961509 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:15.039662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.039567 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702af3e1-839e-4260-b7a1-3f335a313e5b-kube-api-access-cr256\" (UniqueName: \"kubernetes.io/projected/702af3e1-839e-4260-b7a1-3f335a313e5b-kube-api-access-cr256\") pod \"702af3e1-839e-4260-b7a1-3f335a313e5b\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " Nov 03 22:49:15.039842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.039707 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-openshift-ca\") pod \"702af3e1-839e-4260-b7a1-3f335a313e5b\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " Nov 03 22:49:15.039909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.039864 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-xks-ca\") pod \"702af3e1-839e-4260-b7a1-3f335a313e5b\" (UID: \"702af3e1-839e-4260-b7a1-3f335a313e5b\") " Nov 03 22:49:15.040127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.040096 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-openshift-ca" pod "702af3e1-839e-4260-b7a1-3f335a313e5b" (UID: "702af3e1-839e-4260-b7a1-3f335a313e5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:15.040752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.040725 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.042476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.042452 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-xks-ca" pod "702af3e1-839e-4260-b7a1-3f335a313e5b" (UID: "702af3e1-839e-4260-b7a1-3f335a313e5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:49:15.042607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.042485 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702af3e1-839e-4260-b7a1-3f335a313e5b-kube-api-access-cr256" pod "702af3e1-839e-4260-b7a1-3f335a313e5b" (UID: "702af3e1-839e-4260-b7a1-3f335a313e5b"). InnerVolumeSpecName "kube-api-access-cr256". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:15.142006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.141975 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr256\" (UniqueName: \"kubernetes.io/projected/702af3e1-839e-4260-b7a1-3f335a313e5b-kube-api-access-cr256\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.142006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.142006 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702af3e1-839e-4260-b7a1-3f335a313e5b-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.382341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.382316 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:49:15.446522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.446485 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-istio-token\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-istio-token\") pod \"967c277a-6315-4353-8e39-b05adb456f1d\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " Nov 03 22:49:15.446716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.446557 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-certs\") pod \"967c277a-6315-4353-8e39-b05adb456f1d\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " Nov 03 22:49:15.446716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.446637 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-data\") pod \"967c277a-6315-4353-8e39-b05adb456f1d\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " Nov 03 22:49:15.446716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.446662 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-envoy\") pod \"967c277a-6315-4353-8e39-b05adb456f1d\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " Nov 03 22:49:15.446716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.446697 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967c277a-6315-4353-8e39-b05adb456f1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967c277a-6315-4353-8e39-b05adb456f1d-istiod-ca-cert\") pod \"967c277a-6315-4353-8e39-b05adb456f1d\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " Nov 03 22:49:15.446964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.446805 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/967c277a-6315-4353-8e39-b05adb456f1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967c277a-6315-4353-8e39-b05adb456f1d-istio-podinfo\") pod \"967c277a-6315-4353-8e39-b05adb456f1d\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " Nov 03 22:49:15.446964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.446927 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-credential-socket\") pod \"967c277a-6315-4353-8e39-b05adb456f1d\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " Nov 03 22:49:15.446964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.446928 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-certs" pod "967c277a-6315-4353-8e39-b05adb456f1d" (UID: "967c277a-6315-4353-8e39-b05adb456f1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:15.447127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.446999 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-kube-api-access-7s8f4\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-kube-api-access-7s8f4\") pod \"967c277a-6315-4353-8e39-b05adb456f1d\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " Nov 03 22:49:15.447127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.447057 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-socket\") pod \"967c277a-6315-4353-8e39-b05adb456f1d\" (UID: \"967c277a-6315-4353-8e39-b05adb456f1d\") " Nov 03 22:49:15.447852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.447826 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.447992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.447888 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967c277a-6315-4353-8e39-b05adb456f1d-istiod-ca-cert" pod "967c277a-6315-4353-8e39-b05adb456f1d" (UID: "967c277a-6315-4353-8e39-b05adb456f1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:49:15.448155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.448129 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-data" pod "967c277a-6315-4353-8e39-b05adb456f1d" (UID: "967c277a-6315-4353-8e39-b05adb456f1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:15.448253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.448193 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-socket" pod "967c277a-6315-4353-8e39-b05adb456f1d" (UID: "967c277a-6315-4353-8e39-b05adb456f1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:15.448588 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.448564 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-credential-socket" pod "967c277a-6315-4353-8e39-b05adb456f1d" (UID: "967c277a-6315-4353-8e39-b05adb456f1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:15.449194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.449171 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-istio-token" pod "967c277a-6315-4353-8e39-b05adb456f1d" (UID: "967c277a-6315-4353-8e39-b05adb456f1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:15.449794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.449767 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-envoy" pod "967c277a-6315-4353-8e39-b05adb456f1d" (UID: "967c277a-6315-4353-8e39-b05adb456f1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:49:15.450796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.450771 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-kube-api-access-7s8f4" pod "967c277a-6315-4353-8e39-b05adb456f1d" (UID: "967c277a-6315-4353-8e39-b05adb456f1d"). InnerVolumeSpecName "kube-api-access-7s8f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:49:15.451009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.450989 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/967c277a-6315-4353-8e39-b05adb456f1d-istio-podinfo" pod "967c277a-6315-4353-8e39-b05adb456f1d" (UID: "967c277a-6315-4353-8e39-b05adb456f1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:49:15.549127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.549091 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.549127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.549121 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8f4\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-kube-api-access-7s8f4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.549127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.549135 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.549452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.549144 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/967c277a-6315-4353-8e39-b05adb456f1d-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.549452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.549153 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.549452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.549161 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/967c277a-6315-4353-8e39-b05adb456f1d-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.549452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.549172 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/967c277a-6315-4353-8e39-b05adb456f1d-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.549452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.549185 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/967c277a-6315-4353-8e39-b05adb456f1d-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:49:15.765551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.765525 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" Nov 03 22:49:15.765701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.765557 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp" event={"ID":"702af3e1-839e-4260-b7a1-3f335a313e5b","Type":"ContainerDied","Data":"dd53d69195041fb07672cf762d2b3d31da231dffe8b4afdc0177bbd4746f5c50"} Nov 03 22:49:15.765701 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.765620 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd53d69195041fb07672cf762d2b3d31da231dffe8b4afdc0177bbd4746f5c50" Nov 03 22:49:15.767798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.767768 4903 generic.go:356] "Generic (PLEG): container finished" podID="967c277a-6315-4353-8e39-b05adb456f1d" containerID="035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4" exitCode=0 Nov 03 22:49:15.767919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.767839 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" Nov 03 22:49:15.768050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.768021 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" event={"ID":"967c277a-6315-4353-8e39-b05adb456f1d","Type":"ContainerDied","Data":"035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4"} Nov 03 22:49:15.768164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.768089 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q" event={"ID":"967c277a-6315-4353-8e39-b05adb456f1d","Type":"ContainerDied","Data":"2165aecb2fe1e9840527eb8974d17ee40e0b2037bf9500cf1c982b7cf03ce9d4"} Nov 03 22:49:15.768164 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.768122 4903 scope.go:122] "RemoveContainer" containerID="035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4" Nov 03 22:49:15.782733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.782715 4903 scope.go:122] "RemoveContainer" containerID="8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0" Nov 03 22:49:15.793989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.793969 4903 scope.go:122] "RemoveContainer" containerID="035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4" Nov 03 22:49:15.794310 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:49:15.794276 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4\": container with ID starting with 035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4 not found: ID does not exist" containerID="035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4" Nov 03 22:49:15.794367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.794319 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4"} err="failed to get container status \"035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4\": rpc error: code = NotFound desc = could not find container \"035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4\": container with ID starting with 035c437232234ee939aaffdbad4038d6d63ed86a28f8c289722c5a32ecc857c4 not found: ID does not exist" Nov 03 22:49:15.794367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.794341 4903 scope.go:122] "RemoveContainer" containerID="8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0" Nov 03 22:49:15.794599 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:49:15.794582 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0\": container with ID starting with 8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0 not found: ID does not exist" containerID="8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0" Nov 03 22:49:15.794643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.794608 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0"} err="failed to get container status \"8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0\": rpc error: code = NotFound desc = could not find container \"8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0\": container with ID starting with 8c07ac875ff7a498e7446886dad19baa9bbb362275617097b002ee18327abfb0 not found: ID does not exist" Nov 03 22:49:15.827716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.827683 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q"] Nov 03 22:49:15.835028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:15.835002 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9e540d-openshift-default-74d865bc89-p5z9q"] Nov 03 22:49:17.735985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:17.735952 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967c277a-6315-4353-8e39-b05adb456f1d" path="/var/lib/kubelet/pods/967c277a-6315-4353-8e39-b05adb456f1d/volumes" Nov 03 22:49:18.385350 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:49:18.385287 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:49:31.052656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:31.052605 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf"] Nov 03 22:49:31.057881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:31.057854 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4512c7f-ctzhf"] Nov 03 22:49:31.772383 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:31.770740 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c41be0-54fb-458c-9c04-6096c4e589ba" path="/var/lib/kubelet/pods/08c41be0-54fb-458c-9c04-6096c4e589ba/volumes" Nov 03 22:49:48.385829 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:49:48.385789 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:49:53.039452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:53.039405 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp"] Nov 03 22:49:53.075116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:53.075081 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a0e6815-s22hp"] Nov 03 22:49:53.521873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:53.521836 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp"] Nov 03 22:49:53.531686 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:53.531654 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-71a7df-tb7qp"] Nov 03 22:49:53.773949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:53.773864 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300945fb-de39-48f9-982f-566890adfb48" path="/var/lib/kubelet/pods/300945fb-de39-48f9-982f-566890adfb48/volumes" Nov 03 22:49:53.775180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:49:53.775154 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702af3e1-839e-4260-b7a1-3f335a313e5b" path="/var/lib/kubelet/pods/702af3e1-839e-4260-b7a1-3f335a313e5b/volumes" Nov 03 22:50:00.734272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:00.734232 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778"] Nov 03 22:50:00.734852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:00.734495 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" containerID="cri-o://db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28" gracePeriod=30 Nov 03 22:50:01.328689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:01.328649 4903 patch_prober.go:29] interesting pod/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:01.328891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:01.328766 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:06.059760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.059732 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:50:06.183861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.183826 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-envoy\") pod \"afe39d68-ed57-4205-adea-4098787dae30\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " Nov 03 22:50:06.184049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.183999 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe39d68-ed57-4205-adea-4098787dae30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe39d68-ed57-4205-adea-4098787dae30-istiod-ca-cert\") pod \"afe39d68-ed57-4205-adea-4098787dae30\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " Nov 03 22:50:06.184049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184032 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-certs\") pod \"afe39d68-ed57-4205-adea-4098787dae30\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " Nov 03 22:50:06.184176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184059 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-istio-token\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-istio-token\") pod \"afe39d68-ed57-4205-adea-4098787dae30\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " Nov 03 22:50:06.184242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184184 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afe39d68-ed57-4205-adea-4098787dae30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe39d68-ed57-4205-adea-4098787dae30-istio-podinfo\") pod \"afe39d68-ed57-4205-adea-4098787dae30\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " Nov 03 22:50:06.184317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184252 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-credential-socket\") pod \"afe39d68-ed57-4205-adea-4098787dae30\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " Nov 03 22:50:06.184317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184286 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-data\") pod \"afe39d68-ed57-4205-adea-4098787dae30\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " Nov 03 22:50:06.184445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184370 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-certs" pod "afe39d68-ed57-4205-adea-4098787dae30" (UID: "afe39d68-ed57-4205-adea-4098787dae30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.184445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184387 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-socket\") pod \"afe39d68-ed57-4205-adea-4098787dae30\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " Nov 03 22:50:06.184445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184430 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-kube-api-access-bmjvm\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-kube-api-access-bmjvm\") pod \"afe39d68-ed57-4205-adea-4098787dae30\" (UID: \"afe39d68-ed57-4205-adea-4098787dae30\") " Nov 03 22:50:06.184638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184618 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-credential-socket" pod "afe39d68-ed57-4205-adea-4098787dae30" (UID: "afe39d68-ed57-4205-adea-4098787dae30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.184708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184522 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe39d68-ed57-4205-adea-4098787dae30-istiod-ca-cert" pod "afe39d68-ed57-4205-adea-4098787dae30" (UID: "afe39d68-ed57-4205-adea-4098787dae30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:06.184825 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184808 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-data" pod "afe39d68-ed57-4205-adea-4098787dae30" (UID: "afe39d68-ed57-4205-adea-4098787dae30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.184890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.184823 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-socket" pod "afe39d68-ed57-4205-adea-4098787dae30" (UID: "afe39d68-ed57-4205-adea-4098787dae30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.185256 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.185228 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe39d68-ed57-4205-adea-4098787dae30-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.185422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.185261 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.185422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.185275 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.185422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.185288 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.185422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.185343 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.186553 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.186526 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-istio-token" pod "afe39d68-ed57-4205-adea-4098787dae30" (UID: "afe39d68-ed57-4205-adea-4098787dae30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:06.186945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.186924 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-envoy" pod "afe39d68-ed57-4205-adea-4098787dae30" (UID: "afe39d68-ed57-4205-adea-4098787dae30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:06.187073 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.187057 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afe39d68-ed57-4205-adea-4098787dae30-istio-podinfo" pod "afe39d68-ed57-4205-adea-4098787dae30" (UID: "afe39d68-ed57-4205-adea-4098787dae30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:06.187205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.187184 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-kube-api-access-bmjvm" pod "afe39d68-ed57-4205-adea-4098787dae30" (UID: "afe39d68-ed57-4205-adea-4098787dae30"). InnerVolumeSpecName "kube-api-access-bmjvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:06.286450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.286369 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe39d68-ed57-4205-adea-4098787dae30-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.286450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.286398 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.286450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.286407 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe39d68-ed57-4205-adea-4098787dae30-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.286450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.286416 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmjvm\" (UniqueName: \"kubernetes.io/projected/afe39d68-ed57-4205-adea-4098787dae30-kube-api-access-bmjvm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:06.310764 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.310729 4903 generic.go:356] "Generic (PLEG): container finished" podID="afe39d68-ed57-4205-adea-4098787dae30" containerID="db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28" exitCode=0 Nov 03 22:50:06.310985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.310774 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" Nov 03 22:50:06.310985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.310813 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" event={"ID":"afe39d68-ed57-4205-adea-4098787dae30","Type":"ContainerDied","Data":"db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28"} Nov 03 22:50:06.310985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.310877 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778" event={"ID":"afe39d68-ed57-4205-adea-4098787dae30","Type":"ContainerDied","Data":"e871f2abdd133a2946262e5d14fe30ba1a677f8aa0202e9ed9ef08d803dd6237"} Nov 03 22:50:06.310985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.310904 4903 scope.go:122] "RemoveContainer" containerID="db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28" Nov 03 22:50:06.324382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.324354 4903 scope.go:122] "RemoveContainer" containerID="db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28" Nov 03 22:50:06.324722 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:50:06.324700 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28\": container with ID starting with db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28 not found: ID does not exist" containerID="db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28" Nov 03 22:50:06.324810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.324733 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28"} err="failed to get container status \"db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28\": rpc error: code = NotFound desc = could not find container \"db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28\": container with ID starting with db6cac190b2325f74980d2ec764f7fb3ed1956576dd4d2ce1d0040a8c5257b28 not found: ID does not exist" Nov 03 22:50:06.363841 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.363808 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778"] Nov 03 22:50:06.369103 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:06.369069 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-71a7df-openshift-default-74d8445657-fv778"] Nov 03 22:50:07.792596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:07.790666 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe39d68-ed57-4205-adea-4098787dae30" path="/var/lib/kubelet/pods/afe39d68-ed57-4205-adea-4098787dae30/volumes" Nov 03 22:50:18.268902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:18.268867 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d"] Nov 03 22:50:18.269356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:18.269099 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" containerID="cri-o://7f951ce8f58e0a974472f7d8d9b5ae98005d4b9443ce9befeee085a31e7be08a" gracePeriod=30 Nov 03 22:50:18.387158 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:50:18.387120 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:50:20.001242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:20.001203 4903 patch_prober.go:29] interesting pod/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:50:20.001783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:20.001279 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:50:23.463488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.459919 4903 generic.go:356] "Generic (PLEG): container finished" podID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerID="7f951ce8f58e0a974472f7d8d9b5ae98005d4b9443ce9befeee085a31e7be08a" exitCode=0 Nov 03 22:50:23.463488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.460241 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" event={"ID":"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575","Type":"ContainerDied","Data":"7f951ce8f58e0a974472f7d8d9b5ae98005d4b9443ce9befeee085a31e7be08a"} Nov 03 22:50:23.568841 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.568808 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:50:23.669008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.668973 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-envoy\") pod \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " Nov 03 22:50:23.669200 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.669038 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-podinfo\") pod \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " Nov 03 22:50:23.669200 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.669105 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-socket\") pod \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " Nov 03 22:50:23.669200 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.669174 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-token\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-token\") pod \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " Nov 03 22:50:23.669658 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.669633 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-certs\") pod \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " Nov 03 22:50:23.669867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.669850 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-kube-api-access-dczqd\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-kube-api-access-dczqd\") pod \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " Nov 03 22:50:23.670098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670073 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-data\") pod \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " Nov 03 22:50:23.670098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.669928 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-certs" pod "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" (UID: "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.670279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670080 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-socket" pod "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" (UID: "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.670279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670117 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-credential-socket\") pod \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " Nov 03 22:50:23.670279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670262 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-data" pod "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" (UID: "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.670279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670262 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istiod-ca-cert\") pod \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\" (UID: \"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575\") " Nov 03 22:50:23.670555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670406 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-credential-socket" pod "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" (UID: "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.670620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670585 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istiod-ca-cert" pod "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" (UID: "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:50:23.670965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670944 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.670965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670965 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.671100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670974 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.671100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670981 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.671100 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.670989 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.672449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.672422 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-envoy" pod "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" (UID: "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:50:23.672449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.672433 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-podinfo" pod "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" (UID: "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:50:23.673184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.673157 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-token" pod "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" (UID: "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:23.673270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.673173 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-kube-api-access-dczqd" pod "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" (UID: "a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575"). InnerVolumeSpecName "kube-api-access-dczqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:50:23.772941 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.772904 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.773110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.772953 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dczqd\" (UniqueName: \"kubernetes.io/projected/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-kube-api-access-dczqd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.773110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.772974 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:23.773110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:23.772988 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:50:24.470927 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:24.470901 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" Nov 03 22:50:24.471481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:24.470932 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d" event={"ID":"a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575","Type":"ContainerDied","Data":"f4a2eba08fda798b70fcdaf8f04912d28d94a47de93fadaae3e329b6a371ac42"} Nov 03 22:50:24.471481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:24.471000 4903 scope.go:122] "RemoveContainer" containerID="7f951ce8f58e0a974472f7d8d9b5ae98005d4b9443ce9befeee085a31e7be08a" Nov 03 22:50:24.530528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:24.530496 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d"] Nov 03 22:50:24.543801 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:24.543769 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-553753-openshift-default-558949869c-ktl7d"] Nov 03 22:50:25.735444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:25.735405 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" path="/var/lib/kubelet/pods/a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575/volumes" Nov 03 22:50:30.903316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:30.898639 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:50:48.396217 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:50:48.396174 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:50:57.233311 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.233260 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm"] Nov 03 22:50:57.234520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234499 4903 cpu_manager.go:414] "containerMap: removing container" podUID="702af3e1-839e-4260-b7a1-3f335a313e5b" containerName="revoke-keys" Nov 03 22:50:57.234610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234523 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702af3e1-839e-4260-b7a1-3f335a313e5b" containerName="revoke-keys" Nov 03 22:50:57.234610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234549 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" Nov 03 22:50:57.234610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234559 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" Nov 03 22:50:57.234742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234609 4903 cpu_manager.go:414] "containerMap: removing container" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" Nov 03 22:50:57.234742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234619 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" Nov 03 22:50:57.234742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234653 4903 cpu_manager.go:414] "containerMap: removing container" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" Nov 03 22:50:57.234742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234662 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" Nov 03 22:50:57.234742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234703 4903 cpu_manager.go:414] "containerMap: removing container" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" Nov 03 22:50:57.234742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234711 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" Nov 03 22:50:57.234990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234884 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a167e1e8-f6f8-4df5-87e6-5fc3d8d3c575" containerName="istio-proxy" Nov 03 22:50:57.234990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234908 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe39d68-ed57-4205-adea-4098787dae30" containerName="istio-proxy" Nov 03 22:50:57.234990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234923 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" Nov 03 22:50:57.234990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234939 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="967c277a-6315-4353-8e39-b05adb456f1d" containerName="istio-proxy" Nov 03 22:50:57.234990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.234952 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="702af3e1-839e-4260-b7a1-3f335a313e5b" containerName="revoke-keys" Nov 03 22:50:57.239949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.239925 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.253768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.253742 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm"] Nov 03 22:50:57.256003 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.255983 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:50:57.256154 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.256006 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:50:57.259550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.259528 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:50:57.259659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.259552 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:50:57.339470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.339412 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf"] Nov 03 22:50:57.353717 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.353586 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.386364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.385556 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf"] Nov 03 22:50:57.412032 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.411994 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13680af1-3551-4e3c-83fd-385621682d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af6c3b07-f9xvm\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.412433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.412405 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcv49\" (UniqueName: \"kubernetes.io/projected/13680af1-3551-4e3c-83fd-385621682d25-kube-api-access-gcv49\") pod \"maas-api-revoke-sub-af6c3b07-f9xvm\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.412583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.412503 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13680af1-3551-4e3c-83fd-385621682d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af6c3b07-f9xvm\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.514357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.514233 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcv49\" (UniqueName: \"kubernetes.io/projected/13680af1-3551-4e3c-83fd-385621682d25-kube-api-access-gcv49\") pod \"maas-api-revoke-sub-af6c3b07-f9xvm\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.514357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.514354 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13680af1-3551-4e3c-83fd-385621682d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af6c3b07-f9xvm\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.514576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.514376 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5767a5-psrqf\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.514576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.514482 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5767a5-psrqf\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.514660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.514643 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13680af1-3551-4e3c-83fd-385621682d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af6c3b07-f9xvm\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.514719 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.514703 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lmg\" (UniqueName: \"kubernetes.io/projected/789d7855-e65b-4f71-9ce6-f053a2500e98-kube-api-access-f4lmg\") pod \"maas-api-revoke-sub-4f5767a5-psrqf\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.515354 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.515331 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13680af1-3551-4e3c-83fd-385621682d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af6c3b07-f9xvm\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.516964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.516942 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13680af1-3551-4e3c-83fd-385621682d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af6c3b07-f9xvm\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.535819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.535786 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcv49\" (UniqueName: \"kubernetes.io/projected/13680af1-3551-4e3c-83fd-385621682d25-kube-api-access-gcv49\") pod \"maas-api-revoke-sub-af6c3b07-f9xvm\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.559821 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.559791 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:50:57.626348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.618090 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lmg\" (UniqueName: \"kubernetes.io/projected/789d7855-e65b-4f71-9ce6-f053a2500e98-kube-api-access-f4lmg\") pod \"maas-api-revoke-sub-4f5767a5-psrqf\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.626348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.618721 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5767a5-psrqf\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.626348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.618909 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5767a5-psrqf\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.626348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.619695 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5767a5-psrqf\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.628664 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.628639 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5767a5-psrqf\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.640447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.640415 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lmg\" (UniqueName: \"kubernetes.io/projected/789d7855-e65b-4f71-9ce6-f053a2500e98-kube-api-access-f4lmg\") pod \"maas-api-revoke-sub-4f5767a5-psrqf\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.691391 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.691354 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:50:57.852573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.848450 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl"] Nov 03 22:50:57.852573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.848795 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" containerID="cri-o://c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874" gracePeriod=30 Nov 03 22:50:57.882088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:57.882053 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm"] Nov 03 22:50:58.016804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:58.016748 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf"] Nov 03 22:50:58.017045 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:50:58.017026 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789d7855_e65b_4f71_9ce6_f053a2500e98.slice/crio-59e1c080d442c440339c198d984ced85f929988409c08c85c7f6eb0edf8b2cf5 WatchSource:0}: Error finding container 59e1c080d442c440339c198d984ced85f929988409c08c85c7f6eb0edf8b2cf5: Status 404 returned error can't find the container with id 59e1c080d442c440339c198d984ced85f929988409c08c85c7f6eb0edf8b2cf5 Nov 03 22:50:58.674249 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:58.674218 4903 scope.go:122] "RemoveContainer" containerID="eb69a4883811092986f6f82b2a2fa2a7f2c8ba37b87932b25b20ad74cecd8370" Nov 03 22:50:58.685113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:58.685087 4903 scope.go:122] "RemoveContainer" containerID="a41f82e17a7da479a33f754f6251e9a8a63fc3a4bab77acb7e8e39e06126ff9d" Nov 03 22:50:58.828039 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:58.828002 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" event={"ID":"789d7855-e65b-4f71-9ce6-f053a2500e98","Type":"ContainerStarted","Data":"f1bd3db269231439dfe49097cd91975a4e55426d22b4ae45aed1ca8550c2037a"} Nov 03 22:50:58.830414 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:58.830387 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" event={"ID":"789d7855-e65b-4f71-9ce6-f053a2500e98","Type":"ContainerStarted","Data":"59e1c080d442c440339c198d984ced85f929988409c08c85c7f6eb0edf8b2cf5"} Nov 03 22:50:58.862259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:58.862222 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" event={"ID":"13680af1-3551-4e3c-83fd-385621682d25","Type":"ContainerStarted","Data":"137f7b4dde11509368607b233cd36530ccfb773ef27d817c3e51a0a83b4e1433"} Nov 03 22:50:58.862480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:58.862277 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" event={"ID":"13680af1-3551-4e3c-83fd-385621682d25","Type":"ContainerStarted","Data":"97564317c78247582129ca8a429c6e73538a84e0317eb6bd7725cbac0f9bd9c7"} Nov 03 22:50:58.876900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:58.876819 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" podStartSLOduration=1.876799965 podStartE2EDuration="1.876799965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:58.872360456 +0000 UTC m=+2285.276301433" watchObservedRunningTime="2026-11-03 22:50:58.876799965 +0000 UTC m=+2285.280740977" Nov 03 22:50:58.912449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:58.912394 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" podStartSLOduration=1.91237988 podStartE2EDuration="1.91237988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:50:58.909742973 +0000 UTC m=+2285.313683951" watchObservedRunningTime="2026-11-03 22:50:58.91237988 +0000 UTC m=+2285.316320855" Nov 03 22:50:59.873477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:59.873443 4903 generic.go:356] "Generic (PLEG): container finished" podID="789d7855-e65b-4f71-9ce6-f053a2500e98" containerID="f1bd3db269231439dfe49097cd91975a4e55426d22b4ae45aed1ca8550c2037a" exitCode=0 Nov 03 22:50:59.873988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:59.873535 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" event={"ID":"789d7855-e65b-4f71-9ce6-f053a2500e98","Type":"ContainerDied","Data":"f1bd3db269231439dfe49097cd91975a4e55426d22b4ae45aed1ca8550c2037a"} Nov 03 22:50:59.875395 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:59.875367 4903 generic.go:356] "Generic (PLEG): container finished" podID="13680af1-3551-4e3c-83fd-385621682d25" containerID="137f7b4dde11509368607b233cd36530ccfb773ef27d817c3e51a0a83b4e1433" exitCode=0 Nov 03 22:50:59.875538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:50:59.875456 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" event={"ID":"13680af1-3551-4e3c-83fd-385621682d25","Type":"ContainerDied","Data":"137f7b4dde11509368607b233cd36530ccfb773ef27d817c3e51a0a83b4e1433"} Nov 03 22:51:01.091209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.091180 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:51:01.139572 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.139541 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:51:01.193289 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.193257 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13680af1-3551-4e3c-83fd-385621682d25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13680af1-3551-4e3c-83fd-385621682d25-maas-api-xks-ca\") pod \"13680af1-3551-4e3c-83fd-385621682d25\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " Nov 03 22:51:01.193476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.193337 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789d7855-e65b-4f71-9ce6-f053a2500e98-kube-api-access-f4lmg\" (UniqueName: \"kubernetes.io/projected/789d7855-e65b-4f71-9ce6-f053a2500e98-kube-api-access-f4lmg\") pod \"789d7855-e65b-4f71-9ce6-f053a2500e98\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " Nov 03 22:51:01.193564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.193501 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13680af1-3551-4e3c-83fd-385621682d25-kube-api-access-gcv49\" (UniqueName: \"kubernetes.io/projected/13680af1-3551-4e3c-83fd-385621682d25-kube-api-access-gcv49\") pod \"13680af1-3551-4e3c-83fd-385621682d25\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " Nov 03 22:51:01.193564 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.193525 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-xks-ca\") pod \"789d7855-e65b-4f71-9ce6-f053a2500e98\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " Nov 03 22:51:01.193690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.193635 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-openshift-ca\") pod \"789d7855-e65b-4f71-9ce6-f053a2500e98\" (UID: \"789d7855-e65b-4f71-9ce6-f053a2500e98\") " Nov 03 22:51:01.193765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.193719 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13680af1-3551-4e3c-83fd-385621682d25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13680af1-3551-4e3c-83fd-385621682d25-maas-api-openshift-ca\") pod \"13680af1-3551-4e3c-83fd-385621682d25\" (UID: \"13680af1-3551-4e3c-83fd-385621682d25\") " Nov 03 22:51:01.194250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.194163 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-openshift-ca" pod "789d7855-e65b-4f71-9ce6-f053a2500e98" (UID: "789d7855-e65b-4f71-9ce6-f053a2500e98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:01.194798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.194357 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13680af1-3551-4e3c-83fd-385621682d25-maas-api-openshift-ca" pod "13680af1-3551-4e3c-83fd-385621682d25" (UID: "13680af1-3551-4e3c-83fd-385621682d25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:01.194798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.194737 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:01.194798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.194767 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13680af1-3551-4e3c-83fd-385621682d25-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:01.196011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.195983 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13680af1-3551-4e3c-83fd-385621682d25-kube-api-access-gcv49" pod "13680af1-3551-4e3c-83fd-385621682d25" (UID: "13680af1-3551-4e3c-83fd-385621682d25"). InnerVolumeSpecName "kube-api-access-gcv49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:01.196625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.196557 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789d7855-e65b-4f71-9ce6-f053a2500e98-kube-api-access-f4lmg" pod "789d7855-e65b-4f71-9ce6-f053a2500e98" (UID: "789d7855-e65b-4f71-9ce6-f053a2500e98"). InnerVolumeSpecName "kube-api-access-f4lmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:01.196625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.196581 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-xks-ca" pod "789d7855-e65b-4f71-9ce6-f053a2500e98" (UID: "789d7855-e65b-4f71-9ce6-f053a2500e98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:01.198359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.198321 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13680af1-3551-4e3c-83fd-385621682d25-maas-api-xks-ca" pod "13680af1-3551-4e3c-83fd-385621682d25" (UID: "13680af1-3551-4e3c-83fd-385621682d25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:01.296875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.296839 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13680af1-3551-4e3c-83fd-385621682d25-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:01.296875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.296869 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4lmg\" (UniqueName: \"kubernetes.io/projected/789d7855-e65b-4f71-9ce6-f053a2500e98-kube-api-access-f4lmg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:01.296875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.296880 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcv49\" (UniqueName: \"kubernetes.io/projected/13680af1-3551-4e3c-83fd-385621682d25-kube-api-access-gcv49\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:01.297124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.296890 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/789d7855-e65b-4f71-9ce6-f053a2500e98-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:01.899515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.899483 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" Nov 03 22:51:01.899515 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.899491 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf" event={"ID":"789d7855-e65b-4f71-9ce6-f053a2500e98","Type":"ContainerDied","Data":"59e1c080d442c440339c198d984ced85f929988409c08c85c7f6eb0edf8b2cf5"} Nov 03 22:51:01.899839 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.899544 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e1c080d442c440339c198d984ced85f929988409c08c85c7f6eb0edf8b2cf5" Nov 03 22:51:01.913093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.913050 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" event={"ID":"13680af1-3551-4e3c-83fd-385621682d25","Type":"ContainerDied","Data":"97564317c78247582129ca8a429c6e73538a84e0317eb6bd7725cbac0f9bd9c7"} Nov 03 22:51:01.913309 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.913138 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97564317c78247582129ca8a429c6e73538a84e0317eb6bd7725cbac0f9bd9c7" Nov 03 22:51:01.913309 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:01.913228 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm" Nov 03 22:51:02.919314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:02.919272 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 22:51:02.919733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:02.919341 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 22:51:03.156301 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.156259 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:51:03.232936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.232885 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-socket\") pod \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " Nov 03 22:51:03.233279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.233245 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-socket" pod "d560378b-a5b9-4a00-8f7c-7319c1d1bb87" (UID: "d560378b-a5b9-4a00-8f7c-7319c1d1bb87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:03.233729 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.233689 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-data\") pod \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " Nov 03 22:51:03.233854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.233836 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istiod-ca-cert\") pod \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " Nov 03 22:51:03.233924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.233883 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-certs\") pod \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " Nov 03 22:51:03.233987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.233957 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-credential-socket\") pod \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " Nov 03 22:51:03.234046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.233974 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-data" pod "d560378b-a5b9-4a00-8f7c-7319c1d1bb87" (UID: "d560378b-a5b9-4a00-8f7c-7319c1d1bb87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:03.234046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.234025 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-token\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-token\") pod \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " Nov 03 22:51:03.234159 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.234059 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-envoy\") pod \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " Nov 03 22:51:03.234217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.234157 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-podinfo\") pod \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " Nov 03 22:51:03.234270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.234256 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-kube-api-access-r9tql\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-kube-api-access-r9tql\") pod \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\" (UID: \"d560378b-a5b9-4a00-8f7c-7319c1d1bb87\") " Nov 03 22:51:03.234361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.234337 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istiod-ca-cert" pod "d560378b-a5b9-4a00-8f7c-7319c1d1bb87" (UID: "d560378b-a5b9-4a00-8f7c-7319c1d1bb87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:03.237148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.237114 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-token" pod "d560378b-a5b9-4a00-8f7c-7319c1d1bb87" (UID: "d560378b-a5b9-4a00-8f7c-7319c1d1bb87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:03.237280 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.237152 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-kube-api-access-r9tql" pod "d560378b-a5b9-4a00-8f7c-7319c1d1bb87" (UID: "d560378b-a5b9-4a00-8f7c-7319c1d1bb87"). InnerVolumeSpecName "kube-api-access-r9tql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:03.237406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.237387 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-certs" pod "d560378b-a5b9-4a00-8f7c-7319c1d1bb87" (UID: "d560378b-a5b9-4a00-8f7c-7319c1d1bb87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:03.237599 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.237575 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-credential-socket" pod "d560378b-a5b9-4a00-8f7c-7319c1d1bb87" (UID: "d560378b-a5b9-4a00-8f7c-7319c1d1bb87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:03.238423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.238401 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:03.238520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.238430 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:03.238520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.238444 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:03.238520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.238457 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:03.238520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.238469 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:03.238520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.238481 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:03.238520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.238494 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9tql\" (UniqueName: \"kubernetes.io/projected/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-kube-api-access-r9tql\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:03.242053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.240150 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-envoy" pod "d560378b-a5b9-4a00-8f7c-7319c1d1bb87" (UID: "d560378b-a5b9-4a00-8f7c-7319c1d1bb87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:03.242053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.241269 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-podinfo" pod "d560378b-a5b9-4a00-8f7c-7319c1d1bb87" (UID: "d560378b-a5b9-4a00-8f7c-7319c1d1bb87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:03.340122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.340085 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:03.340122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.340118 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d560378b-a5b9-4a00-8f7c-7319c1d1bb87-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:03.929633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.929591 4903 generic.go:356] "Generic (PLEG): container finished" podID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerID="c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874" exitCode=0 Nov 03 22:51:03.930029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.929641 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" Nov 03 22:51:03.930029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.929677 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" event={"ID":"d560378b-a5b9-4a00-8f7c-7319c1d1bb87","Type":"ContainerDied","Data":"c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874"} Nov 03 22:51:03.930029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.929740 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl" event={"ID":"d560378b-a5b9-4a00-8f7c-7319c1d1bb87","Type":"ContainerDied","Data":"6995fceaf6b63bf645ebabda487a4c848e088b4e390c6bf1b2b905353ea0ee28"} Nov 03 22:51:03.930029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.929765 4903 scope.go:122] "RemoveContainer" containerID="c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874" Nov 03 22:51:03.940954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.940931 4903 scope.go:122] "RemoveContainer" containerID="c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874" Nov 03 22:51:03.941229 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:51:03.941209 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874\": container with ID starting with c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874 not found: ID does not exist" containerID="c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874" Nov 03 22:51:03.941287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.941238 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874"} err="failed to get container status \"c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874\": rpc error: code = NotFound desc = could not find container \"c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874\": container with ID starting with c11754cdc2385e402b4cab507e24601f0fe62d9119fdedb134691d2fdc09c874 not found: ID does not exist" Nov 03 22:51:03.990831 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:03.990799 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl"] Nov 03 22:51:04.000832 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:04.000754 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0847aa-openshift-default-75d5774789-5jtnl"] Nov 03 22:51:05.758633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:05.758591 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" path="/var/lib/kubelet/pods/d560378b-a5b9-4a00-8f7c-7319c1d1bb87/volumes" Nov 03 22:51:09.900768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:09.900720 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2"] Nov 03 22:51:10.440798 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.440761 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt"] Nov 03 22:51:10.441547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.441532 4903 cpu_manager.go:414] "containerMap: removing container" podUID="789d7855-e65b-4f71-9ce6-f053a2500e98" containerName="revoke-keys" Nov 03 22:51:10.441600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.441549 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789d7855-e65b-4f71-9ce6-f053a2500e98" containerName="revoke-keys" Nov 03 22:51:10.441600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.441568 4903 cpu_manager.go:414] "containerMap: removing container" podUID="13680af1-3551-4e3c-83fd-385621682d25" containerName="revoke-keys" Nov 03 22:51:10.441600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.441574 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13680af1-3551-4e3c-83fd-385621682d25" containerName="revoke-keys" Nov 03 22:51:10.441600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.441591 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" Nov 03 22:51:10.441600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.441597 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" Nov 03 22:51:10.441761 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.441728 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="789d7855-e65b-4f71-9ce6-f053a2500e98" containerName="revoke-keys" Nov 03 22:51:10.441761 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.441739 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="13680af1-3551-4e3c-83fd-385621682d25" containerName="revoke-keys" Nov 03 22:51:10.441761 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.441747 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d560378b-a5b9-4a00-8f7c-7319c1d1bb87" containerName="istio-proxy" Nov 03 22:51:10.447251 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.447234 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.462652 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.462622 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:51:10.462811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.462740 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt"] Nov 03 22:51:10.467905 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.467886 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:51:10.468045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.467908 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:51:10.468045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.467957 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:51:10.554420 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.554384 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb188a10-8rpgt\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.554657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.554630 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglfk\" (UniqueName: \"kubernetes.io/projected/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-kube-api-access-lglfk\") pod \"maas-api-revoke-sub-cb188a10-8rpgt\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.554780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.554765 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb188a10-8rpgt\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.656288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.656252 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglfk\" (UniqueName: \"kubernetes.io/projected/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-kube-api-access-lglfk\") pod \"maas-api-revoke-sub-cb188a10-8rpgt\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.656497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.656391 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb188a10-8rpgt\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.656497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.656459 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb188a10-8rpgt\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.657151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.657121 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb188a10-8rpgt\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.658816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.658798 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb188a10-8rpgt\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.677471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.677440 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglfk\" (UniqueName: \"kubernetes.io/projected/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-kube-api-access-lglfk\") pod \"maas-api-revoke-sub-cb188a10-8rpgt\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:10.764543 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:10.764469 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:11.073360 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:11.073332 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt"] Nov 03 22:51:11.073716 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:51:11.073523 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ed1b4e_3f61_407c_87b4_ad7e8ec65a20.slice/crio-9391a0a86f8a32ae531ca2f4b60fe678084f725266ab781b79c487df1bb14a88 WatchSource:0}: Error finding container 9391a0a86f8a32ae531ca2f4b60fe678084f725266ab781b79c487df1bb14a88: Status 404 returned error can't find the container with id 9391a0a86f8a32ae531ca2f4b60fe678084f725266ab781b79c487df1bb14a88 Nov 03 22:51:12.000473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:12.000438 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" event={"ID":"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20","Type":"ContainerStarted","Data":"1ff698031a37f99a5110f77529931915a5315640e7639be582beb1c7936c84f3"} Nov 03 22:51:12.000662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:12.000523 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" event={"ID":"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20","Type":"ContainerStarted","Data":"9391a0a86f8a32ae531ca2f4b60fe678084f725266ab781b79c487df1bb14a88"} Nov 03 22:51:12.054285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:12.054233 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" podStartSLOduration=2.054218138 podStartE2EDuration="2.054218138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:12.049781665 +0000 UTC m=+2298.453722643" watchObservedRunningTime="2026-11-03 22:51:12.054218138 +0000 UTC m=+2298.458159144" Nov 03 22:51:13.011476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:13.011441 4903 generic.go:356] "Generic (PLEG): container finished" podID="53ed1b4e-3f61-407c-87b4-ad7e8ec65a20" containerID="1ff698031a37f99a5110f77529931915a5315640e7639be582beb1c7936c84f3" exitCode=0 Nov 03 22:51:13.011934 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:13.011488 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" event={"ID":"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20","Type":"ContainerDied","Data":"1ff698031a37f99a5110f77529931915a5315640e7639be582beb1c7936c84f3"} Nov 03 22:51:14.211551 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.211523 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:14.268094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.268064 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k"] Nov 03 22:51:14.268266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.268246 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" containerID="cri-o://03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409" gracePeriod=30 Nov 03 22:51:14.308972 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.308887 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-openshift-ca\") pod \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " Nov 03 22:51:14.309151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.309000 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-xks-ca\") pod \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " Nov 03 22:51:14.309151 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.309033 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-kube-api-access-lglfk\" (UniqueName: \"kubernetes.io/projected/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-kube-api-access-lglfk\") pod \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\" (UID: \"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20\") " Nov 03 22:51:14.309344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.309237 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-openshift-ca" pod "53ed1b4e-3f61-407c-87b4-ad7e8ec65a20" (UID: "53ed1b4e-3f61-407c-87b4-ad7e8ec65a20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:14.309845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.309724 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:14.312861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.312833 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-kube-api-access-lglfk" pod "53ed1b4e-3f61-407c-87b4-ad7e8ec65a20" (UID: "53ed1b4e-3f61-407c-87b4-ad7e8ec65a20"). InnerVolumeSpecName "kube-api-access-lglfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:14.317636 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.317608 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-xks-ca" pod "53ed1b4e-3f61-407c-87b4-ad7e8ec65a20" (UID: "53ed1b4e-3f61-407c-87b4-ad7e8ec65a20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:14.411035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.410998 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:14.411035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:14.411028 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lglfk\" (UniqueName: \"kubernetes.io/projected/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20-kube-api-access-lglfk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:15.041902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.041865 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" Nov 03 22:51:15.042105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.041867 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt" event={"ID":"53ed1b4e-3f61-407c-87b4-ad7e8ec65a20","Type":"ContainerDied","Data":"9391a0a86f8a32ae531ca2f4b60fe678084f725266ab781b79c487df1bb14a88"} Nov 03 22:51:15.042105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.042031 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9391a0a86f8a32ae531ca2f4b60fe678084f725266ab781b79c487df1bb14a88" Nov 03 22:51:15.650139 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.650105 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l"] Nov 03 22:51:15.651019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.651000 4903 cpu_manager.go:414] "containerMap: removing container" podUID="53ed1b4e-3f61-407c-87b4-ad7e8ec65a20" containerName="revoke-keys" Nov 03 22:51:15.651066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.651021 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ed1b4e-3f61-407c-87b4-ad7e8ec65a20" containerName="revoke-keys" Nov 03 22:51:15.651169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.651159 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ed1b4e-3f61-407c-87b4-ad7e8ec65a20" containerName="revoke-keys" Nov 03 22:51:15.655480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.655462 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.671717 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.671696 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l"] Nov 03 22:51:15.677079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.677055 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:51:15.677174 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.677100 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:51:15.677322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.677282 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:51:15.677438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.677393 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:51:15.737388 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.737352 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qgn\" (UniqueName: \"kubernetes.io/projected/b4144ae1-a247-4d89-be12-c10a665387ee-kube-api-access-d8qgn\") pod \"maas-api-revoke-sub-53d0fa3f-b7q8l\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.737598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.737530 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53d0fa3f-b7q8l\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.737598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.737571 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53d0fa3f-b7q8l\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.838783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.838745 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53d0fa3f-b7q8l\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.838985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.838793 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53d0fa3f-b7q8l\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.839074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.839059 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qgn\" (UniqueName: \"kubernetes.io/projected/b4144ae1-a247-4d89-be12-c10a665387ee-kube-api-access-d8qgn\") pod \"maas-api-revoke-sub-53d0fa3f-b7q8l\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.839501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.839480 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53d0fa3f-b7q8l\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.841054 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.841035 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53d0fa3f-b7q8l\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.861425 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.861396 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qgn\" (UniqueName: \"kubernetes.io/projected/b4144ae1-a247-4d89-be12-c10a665387ee-kube-api-access-d8qgn\") pod \"maas-api-revoke-sub-53d0fa3f-b7q8l\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:15.970168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:15.970135 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:16.276229 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:51:16.276196 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4144ae1_a247_4d89_be12_c10a665387ee.slice/crio-59df39227cbe2b9644c6dad2ee343abb821c4e5fbbd21c0cfe8145555b0be9d2 WatchSource:0}: Error finding container 59df39227cbe2b9644c6dad2ee343abb821c4e5fbbd21c0cfe8145555b0be9d2: Status 404 returned error can't find the container with id 59df39227cbe2b9644c6dad2ee343abb821c4e5fbbd21c0cfe8145555b0be9d2 Nov 03 22:51:16.276858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:16.276832 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l"] Nov 03 22:51:17.063197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:17.063163 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" event={"ID":"b4144ae1-a247-4d89-be12-c10a665387ee","Type":"ContainerStarted","Data":"bd5b3375b1e05abbe6b85114e470c9194226ef444738be7850f58b1b814f5a59"} Nov 03 22:51:17.063617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:17.063213 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" event={"ID":"b4144ae1-a247-4d89-be12-c10a665387ee","Type":"ContainerStarted","Data":"59df39227cbe2b9644c6dad2ee343abb821c4e5fbbd21c0cfe8145555b0be9d2"} Nov 03 22:51:17.104019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:17.103965 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" podStartSLOduration=2.103948584 podStartE2EDuration="2.103948584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:17.101405252 +0000 UTC m=+2303.505346228" watchObservedRunningTime="2026-11-03 22:51:17.103948584 +0000 UTC m=+2303.507889558" Nov 03 22:51:17.965836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:17.965794 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:17.966060 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:17.965881 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:18.075642 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:18.075608 4903 generic.go:356] "Generic (PLEG): container finished" podID="b4144ae1-a247-4d89-be12-c10a665387ee" containerID="bd5b3375b1e05abbe6b85114e470c9194226ef444738be7850f58b1b814f5a59" exitCode=0 Nov 03 22:51:18.076045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:18.075713 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" event={"ID":"b4144ae1-a247-4d89-be12-c10a665387ee","Type":"ContainerDied","Data":"bd5b3375b1e05abbe6b85114e470c9194226ef444738be7850f58b1b814f5a59"} Nov 03 22:51:18.346936 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:18.346852 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd"] Nov 03 22:51:18.347116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:18.347073 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" containerID="cri-o://17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694" gracePeriod=30 Nov 03 22:51:18.389668 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:51:18.389635 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:51:19.271404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.271378 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:19.298875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.298847 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-openshift-ca\") pod \"b4144ae1-a247-4d89-be12-c10a665387ee\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " Nov 03 22:51:19.299061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.299043 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-xks-ca\") pod \"b4144ae1-a247-4d89-be12-c10a665387ee\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " Nov 03 22:51:19.299165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.299145 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4144ae1-a247-4d89-be12-c10a665387ee-kube-api-access-d8qgn\" (UniqueName: \"kubernetes.io/projected/b4144ae1-a247-4d89-be12-c10a665387ee-kube-api-access-d8qgn\") pod \"b4144ae1-a247-4d89-be12-c10a665387ee\" (UID: \"b4144ae1-a247-4d89-be12-c10a665387ee\") " Nov 03 22:51:19.299254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.299209 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-openshift-ca" pod "b4144ae1-a247-4d89-be12-c10a665387ee" (UID: "b4144ae1-a247-4d89-be12-c10a665387ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:19.300153 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.300125 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.301482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.301454 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4144ae1-a247-4d89-be12-c10a665387ee-kube-api-access-d8qgn" pod "b4144ae1-a247-4d89-be12-c10a665387ee" (UID: "b4144ae1-a247-4d89-be12-c10a665387ee"). InnerVolumeSpecName "kube-api-access-d8qgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:19.301687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.301660 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-xks-ca" pod "b4144ae1-a247-4d89-be12-c10a665387ee" (UID: "b4144ae1-a247-4d89-be12-c10a665387ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:19.401336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.401274 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4144ae1-a247-4d89-be12-c10a665387ee-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.401336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.401317 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8qgn\" (UniqueName: \"kubernetes.io/projected/b4144ae1-a247-4d89-be12-c10a665387ee-kube-api-access-d8qgn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.542216 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.542190 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:51:19.603820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.603783 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-socket\") pod \"ab9993f4-b036-42b2-a58c-fe4264986bf9\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " Nov 03 22:51:19.603985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.603849 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-certs\") pod \"ab9993f4-b036-42b2-a58c-fe4264986bf9\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " Nov 03 22:51:19.603985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.603951 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-podinfo\") pod \"ab9993f4-b036-42b2-a58c-fe4264986bf9\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " Nov 03 22:51:19.604130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604008 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-token\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-token\") pod \"ab9993f4-b036-42b2-a58c-fe4264986bf9\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " Nov 03 22:51:19.604130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604098 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-socket" pod "ab9993f4-b036-42b2-a58c-fe4264986bf9" (UID: "ab9993f4-b036-42b2-a58c-fe4264986bf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:19.604243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604141 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-envoy\") pod \"ab9993f4-b036-42b2-a58c-fe4264986bf9\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " Nov 03 22:51:19.604243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604184 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-data\") pod \"ab9993f4-b036-42b2-a58c-fe4264986bf9\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " Nov 03 22:51:19.604243 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604213 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-certs" pod "ab9993f4-b036-42b2-a58c-fe4264986bf9" (UID: "ab9993f4-b036-42b2-a58c-fe4264986bf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:19.604446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604249 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-credential-socket\") pod \"ab9993f4-b036-42b2-a58c-fe4264986bf9\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " Nov 03 22:51:19.604446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604321 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9993f4-b036-42b2-a58c-fe4264986bf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9993f4-b036-42b2-a58c-fe4264986bf9-istiod-ca-cert\") pod \"ab9993f4-b036-42b2-a58c-fe4264986bf9\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " Nov 03 22:51:19.604446 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604345 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-kube-api-access-7gcr4\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-kube-api-access-7gcr4\") pod \"ab9993f4-b036-42b2-a58c-fe4264986bf9\" (UID: \"ab9993f4-b036-42b2-a58c-fe4264986bf9\") " Nov 03 22:51:19.604898 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604642 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-credential-socket" pod "ab9993f4-b036-42b2-a58c-fe4264986bf9" (UID: "ab9993f4-b036-42b2-a58c-fe4264986bf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:19.604898 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.604815 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9993f4-b036-42b2-a58c-fe4264986bf9-istiod-ca-cert" pod "ab9993f4-b036-42b2-a58c-fe4264986bf9" (UID: "ab9993f4-b036-42b2-a58c-fe4264986bf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:19.605187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.605160 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.605266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.605192 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.605266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.605208 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.605266 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.605221 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9993f4-b036-42b2-a58c-fe4264986bf9-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.605487 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.605326 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-data" pod "ab9993f4-b036-42b2-a58c-fe4264986bf9" (UID: "ab9993f4-b036-42b2-a58c-fe4264986bf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:19.606988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.606965 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-podinfo" pod "ab9993f4-b036-42b2-a58c-fe4264986bf9" (UID: "ab9993f4-b036-42b2-a58c-fe4264986bf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:19.607105 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.606981 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-envoy" pod "ab9993f4-b036-42b2-a58c-fe4264986bf9" (UID: "ab9993f4-b036-42b2-a58c-fe4264986bf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:19.607222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.607114 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-token" pod "ab9993f4-b036-42b2-a58c-fe4264986bf9" (UID: "ab9993f4-b036-42b2-a58c-fe4264986bf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:19.607562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.607536 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-kube-api-access-7gcr4" pod "ab9993f4-b036-42b2-a58c-fe4264986bf9" (UID: "ab9993f4-b036-42b2-a58c-fe4264986bf9"). InnerVolumeSpecName "kube-api-access-7gcr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:19.728448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.728271 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.728448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.728322 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.728448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.728339 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gcr4\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-kube-api-access-7gcr4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.728448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.728354 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:19.728448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:19.728368 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9993f4-b036-42b2-a58c-fe4264986bf9-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:20.106005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.105915 4903 generic.go:356] "Generic (PLEG): container finished" podID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerID="03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409" exitCode=0 Nov 03 22:51:20.106005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.105966 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" Nov 03 22:51:20.106237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.106004 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" event={"ID":"ab9993f4-b036-42b2-a58c-fe4264986bf9","Type":"ContainerDied","Data":"03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409"} Nov 03 22:51:20.106237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.106066 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k" event={"ID":"ab9993f4-b036-42b2-a58c-fe4264986bf9","Type":"ContainerDied","Data":"1425468c355a16966b46b1283efaa93a5053b8e3263f92ee18208afaf4f757b5"} Nov 03 22:51:20.106237 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.106092 4903 scope.go:122] "RemoveContainer" containerID="03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409" Nov 03 22:51:20.107816 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.107800 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" Nov 03 22:51:20.107948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.107860 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l" event={"ID":"b4144ae1-a247-4d89-be12-c10a665387ee","Type":"ContainerDied","Data":"59df39227cbe2b9644c6dad2ee343abb821c4e5fbbd21c0cfe8145555b0be9d2"} Nov 03 22:51:20.107948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.107897 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59df39227cbe2b9644c6dad2ee343abb821c4e5fbbd21c0cfe8145555b0be9d2" Nov 03 22:51:20.118883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.118860 4903 scope.go:122] "RemoveContainer" containerID="03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409" Nov 03 22:51:20.119826 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:51:20.119804 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409\": container with ID starting with 03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409 not found: ID does not exist" containerID="03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409" Nov 03 22:51:20.119914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.119833 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409"} err="failed to get container status \"03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409\": rpc error: code = NotFound desc = could not find container \"03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409\": container with ID starting with 03f5218b49eb6f7909661b1f0bb865b91f278e739d506ecf0375c0ad7d40d409 not found: ID does not exist" Nov 03 22:51:20.170772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.170717 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k"] Nov 03 22:51:20.174685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.174660 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a940d5-openshift-default-767f4ff56c-ksw7k"] Nov 03 22:51:20.778112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.778075 4903 patch_prober.go:29] interesting pod/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:51:20.778570 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:20.778141 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:51:21.738881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:21.738835 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" path="/var/lib/kubelet/pods/ab9993f4-b036-42b2-a58c-fe4264986bf9/volumes" Nov 03 22:51:23.639483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.639404 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:51:23.788876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.788844 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-kube-api-access-b9h8p\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-kube-api-access-b9h8p\") pod \"d5da792a-d829-4789-b232-ab527d42ac6e\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " Nov 03 22:51:23.789064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.788903 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-certs\") pod \"d5da792a-d829-4789-b232-ab527d42ac6e\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " Nov 03 22:51:23.789064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.788924 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-credential-socket\") pod \"d5da792a-d829-4789-b232-ab527d42ac6e\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " Nov 03 22:51:23.789064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789027 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-istio-token\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-istio-token\") pod \"d5da792a-d829-4789-b232-ab527d42ac6e\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " Nov 03 22:51:23.789262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789111 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-envoy\") pod \"d5da792a-d829-4789-b232-ab527d42ac6e\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " Nov 03 22:51:23.789262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789168 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-data\") pod \"d5da792a-d829-4789-b232-ab527d42ac6e\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " Nov 03 22:51:23.789262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789205 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5da792a-d829-4789-b232-ab527d42ac6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5da792a-d829-4789-b232-ab527d42ac6e-istio-podinfo\") pod \"d5da792a-d829-4789-b232-ab527d42ac6e\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " Nov 03 22:51:23.789262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789191 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-certs" pod "d5da792a-d829-4789-b232-ab527d42ac6e" (UID: "d5da792a-d829-4789-b232-ab527d42ac6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:23.789262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789239 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-credential-socket" pod "d5da792a-d829-4789-b232-ab527d42ac6e" (UID: "d5da792a-d829-4789-b232-ab527d42ac6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:23.789565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789377 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-socket\") pod \"d5da792a-d829-4789-b232-ab527d42ac6e\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " Nov 03 22:51:23.789565 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789496 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5da792a-d829-4789-b232-ab527d42ac6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5da792a-d829-4789-b232-ab527d42ac6e-istiod-ca-cert\") pod \"d5da792a-d829-4789-b232-ab527d42ac6e\" (UID: \"d5da792a-d829-4789-b232-ab527d42ac6e\") " Nov 03 22:51:23.789681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789492 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-data" pod "d5da792a-d829-4789-b232-ab527d42ac6e" (UID: "d5da792a-d829-4789-b232-ab527d42ac6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:23.789765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789671 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-socket" pod "d5da792a-d829-4789-b232-ab527d42ac6e" (UID: "d5da792a-d829-4789-b232-ab527d42ac6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:23.789884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.789852 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5da792a-d829-4789-b232-ab527d42ac6e-istiod-ca-cert" pod "d5da792a-d829-4789-b232-ab527d42ac6e" (UID: "d5da792a-d829-4789-b232-ab527d42ac6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:23.790126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.790107 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:23.790165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.790127 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5da792a-d829-4789-b232-ab527d42ac6e-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:23.790165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.790137 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:23.790165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.790146 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:23.790165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.790154 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:23.791884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.791858 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-envoy" pod "d5da792a-d829-4789-b232-ab527d42ac6e" (UID: "d5da792a-d829-4789-b232-ab527d42ac6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:23.791985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.791900 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-istio-token" pod "d5da792a-d829-4789-b232-ab527d42ac6e" (UID: "d5da792a-d829-4789-b232-ab527d42ac6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:23.791985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.791906 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5da792a-d829-4789-b232-ab527d42ac6e-istio-podinfo" pod "d5da792a-d829-4789-b232-ab527d42ac6e" (UID: "d5da792a-d829-4789-b232-ab527d42ac6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:51:23.791985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.791935 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-kube-api-access-b9h8p" pod "d5da792a-d829-4789-b232-ab527d42ac6e" (UID: "d5da792a-d829-4789-b232-ab527d42ac6e"). InnerVolumeSpecName "kube-api-access-b9h8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:23.891258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.891178 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5da792a-d829-4789-b232-ab527d42ac6e-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:23.891258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.891205 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9h8p\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-kube-api-access-b9h8p\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:23.891258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.891217 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5da792a-d829-4789-b232-ab527d42ac6e-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:23.891258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:23.891225 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5da792a-d829-4789-b232-ab527d42ac6e-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:24.143910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.143806 4903 generic.go:356] "Generic (PLEG): container finished" podID="d5da792a-d829-4789-b232-ab527d42ac6e" containerID="17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694" exitCode=0 Nov 03 22:51:24.143910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.143852 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" Nov 03 22:51:24.143910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.143885 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" event={"ID":"d5da792a-d829-4789-b232-ab527d42ac6e","Type":"ContainerDied","Data":"17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694"} Nov 03 22:51:24.144209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.143957 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd" event={"ID":"d5da792a-d829-4789-b232-ab527d42ac6e","Type":"ContainerDied","Data":"4701d77f752bf7bb7f04f68d7548d7a66dfb7f81cb166868fa3c045d2a15dfb5"} Nov 03 22:51:24.144209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.143990 4903 scope.go:122] "RemoveContainer" containerID="17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694" Nov 03 22:51:24.156659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.156635 4903 scope.go:122] "RemoveContainer" containerID="17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694" Nov 03 22:51:24.156996 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:51:24.156972 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694\": container with ID starting with 17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694 not found: ID does not exist" containerID="17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694" Nov 03 22:51:24.157082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.157009 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694"} err="failed to get container status \"17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694\": rpc error: code = NotFound desc = could not find container \"17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694\": container with ID starting with 17910966ff5b89f04332e4e70b179e4759cd7f7b661aeda07a0d9d663113f694 not found: ID does not exist" Nov 03 22:51:24.192909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.192879 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd"] Nov 03 22:51:24.202850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.202815 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-50d738-openshift-default-8d6476cbc-nj6zd"] Nov 03 22:51:24.923863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:24.923827 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" podUID="ea71a7f5-2351-4704-80b7-8cb2b943a3ca" containerName="main" containerID="cri-o://5f3e5e1f8c801a493f14785ce06fc94e8be505ef5c06d4fda58dc11b9fc01f51" gracePeriod=45 Nov 03 22:51:25.157950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.157914 4903 generic.go:356] "Generic (PLEG): container finished" podID="ea71a7f5-2351-4704-80b7-8cb2b943a3ca" containerID="5f3e5e1f8c801a493f14785ce06fc94e8be505ef5c06d4fda58dc11b9fc01f51" exitCode=0 Nov 03 22:51:25.158115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.157991 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" event={"ID":"ea71a7f5-2351-4704-80b7-8cb2b943a3ca","Type":"ContainerDied","Data":"5f3e5e1f8c801a493f14785ce06fc94e8be505ef5c06d4fda58dc11b9fc01f51"} Nov 03 22:51:25.205534 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.205507 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:51:25.312320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.312264 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-dshm\") pod \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " Nov 03 22:51:25.312506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.312347 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tmp-dir\") pod \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " Nov 03 22:51:25.312506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.312392 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tls-certs\") pod \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " Nov 03 22:51:25.312506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.312443 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-model-cache\") pod \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " Nov 03 22:51:25.312506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.312477 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-home\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-home\") pod \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " Nov 03 22:51:25.312714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.312608 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tmp-dir" pod "ea71a7f5-2351-4704-80b7-8cb2b943a3ca" (UID: "ea71a7f5-2351-4704-80b7-8cb2b943a3ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:25.312714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.312700 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-kube-api-access-hkncd\") pod \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\" (UID: \"ea71a7f5-2351-4704-80b7-8cb2b943a3ca\") " Nov 03 22:51:25.312849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.312823 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-model-cache" pod "ea71a7f5-2351-4704-80b7-8cb2b943a3ca" (UID: "ea71a7f5-2351-4704-80b7-8cb2b943a3ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:25.312907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.312838 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-home" pod "ea71a7f5-2351-4704-80b7-8cb2b943a3ca" (UID: "ea71a7f5-2351-4704-80b7-8cb2b943a3ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:25.313412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.313390 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:25.313412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.313409 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:25.313412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.313417 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:25.314895 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.314873 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tls-certs" pod "ea71a7f5-2351-4704-80b7-8cb2b943a3ca" (UID: "ea71a7f5-2351-4704-80b7-8cb2b943a3ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:25.314984 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.314911 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-kube-api-access-hkncd" pod "ea71a7f5-2351-4704-80b7-8cb2b943a3ca" (UID: "ea71a7f5-2351-4704-80b7-8cb2b943a3ca"). InnerVolumeSpecName "kube-api-access-hkncd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:25.315422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.315407 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-dshm" pod "ea71a7f5-2351-4704-80b7-8cb2b943a3ca" (UID: "ea71a7f5-2351-4704-80b7-8cb2b943a3ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:25.414605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.414577 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkncd\" (UniqueName: \"kubernetes.io/projected/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-kube-api-access-hkncd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:25.414605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.414603 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:25.414811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.414616 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea71a7f5-2351-4704-80b7-8cb2b943a3ca-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:25.741338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:25.738927 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" path="/var/lib/kubelet/pods/d5da792a-d829-4789-b232-ab527d42ac6e/volumes" Nov 03 22:51:26.171043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:26.171013 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" Nov 03 22:51:26.171525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:26.171038 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2" event={"ID":"ea71a7f5-2351-4704-80b7-8cb2b943a3ca","Type":"ContainerDied","Data":"240d102bc20494a2f077f3aadfcfd95d208adbefd462873e0b81ca8e4ba5ae5d"} Nov 03 22:51:26.171525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:26.171088 4903 scope.go:122] "RemoveContainer" containerID="5f3e5e1f8c801a493f14785ce06fc94e8be505ef5c06d4fda58dc11b9fc01f51" Nov 03 22:51:26.218752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:26.218719 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2"] Nov 03 22:51:26.229220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:26.229195 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7940ea/test-model-7940ea-kserve-7765dc7479-6r5p2"] Nov 03 22:51:27.221571 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.221539 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx"] Nov 03 22:51:27.481744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.481657 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q"] Nov 03 22:51:27.482598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482573 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" Nov 03 22:51:27.482598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482597 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" Nov 03 22:51:27.482814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482645 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b4144ae1-a247-4d89-be12-c10a665387ee" containerName="revoke-keys" Nov 03 22:51:27.482814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482653 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4144ae1-a247-4d89-be12-c10a665387ee" containerName="revoke-keys" Nov 03 22:51:27.482814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482676 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ea71a7f5-2351-4704-80b7-8cb2b943a3ca" containerName="main" Nov 03 22:51:27.482814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482684 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea71a7f5-2351-4704-80b7-8cb2b943a3ca" containerName="main" Nov 03 22:51:27.482814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482709 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" Nov 03 22:51:27.482814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482715 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" Nov 03 22:51:27.483112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482916 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5da792a-d829-4789-b232-ab527d42ac6e" containerName="istio-proxy" Nov 03 22:51:27.483112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482939 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4144ae1-a247-4d89-be12-c10a665387ee" containerName="revoke-keys" Nov 03 22:51:27.483112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482954 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea71a7f5-2351-4704-80b7-8cb2b943a3ca" containerName="main" Nov 03 22:51:27.483112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.482967 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9993f4-b036-42b2-a58c-fe4264986bf9" containerName="istio-proxy" Nov 03 22:51:27.488455 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.488431 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.502445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.501751 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:51:27.504557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.504532 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:51:27.505144 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.505123 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q"] Nov 03 22:51:27.579907 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.579875 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:51:27.580081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.579963 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:51:27.649442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.649403 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.649642 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.649452 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgktp\" (UniqueName: \"kubernetes.io/projected/1eb52cd6-fc70-46e4-8e97-a500d0496c09-kube-api-access-lgktp\") pod \"maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.649642 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.649572 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.757143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.757055 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.758257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.758225 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgktp\" (UniqueName: \"kubernetes.io/projected/1eb52cd6-fc70-46e4-8e97-a500d0496c09-kube-api-access-lgktp\") pod \"maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.758468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.758310 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.766193 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.766154 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.779390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.779359 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea71a7f5-2351-4704-80b7-8cb2b943a3ca" path="/var/lib/kubelet/pods/ea71a7f5-2351-4704-80b7-8cb2b943a3ca/volumes" Nov 03 22:51:27.793262 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.793213 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.850428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.850394 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgktp\" (UniqueName: \"kubernetes.io/projected/1eb52cd6-fc70-46e4-8e97-a500d0496c09-kube-api-access-lgktp\") pod \"maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:27.993754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.993724 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6"] Nov 03 22:51:27.999450 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:27.999424 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.018084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.018054 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6"] Nov 03 22:51:28.107584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.107550 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:28.165934 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.165892 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xsj\" (UniqueName: \"kubernetes.io/projected/e71ad748-a399-48ba-84b5-e9dff218d96f-kube-api-access-s8xsj\") pod \"maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.166222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.166092 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.166222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.166133 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.268147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.268058 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.268147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.268107 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.268618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.268281 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xsj\" (UniqueName: \"kubernetes.io/projected/e71ad748-a399-48ba-84b5-e9dff218d96f-kube-api-access-s8xsj\") pod \"maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.268752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.268732 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.270536 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.270510 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.290048 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.290024 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xsj\" (UniqueName: \"kubernetes.io/projected/e71ad748-a399-48ba-84b5-e9dff218d96f-kube-api-access-s8xsj\") pod \"maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.315873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.315850 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:28.421078 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.421045 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q"] Nov 03 22:51:28.422554 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:51:28.422504 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb52cd6_fc70_46e4_8e97_a500d0496c09.slice/crio-b90410d782022e27117f6c1a34fdf7410bc55f038a75d69fa47ccc7902515cc8 WatchSource:0}: Error finding container b90410d782022e27117f6c1a34fdf7410bc55f038a75d69fa47ccc7902515cc8: Status 404 returned error can't find the container with id b90410d782022e27117f6c1a34fdf7410bc55f038a75d69fa47ccc7902515cc8 Nov 03 22:51:28.621306 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:28.621266 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6"] Nov 03 22:51:28.621714 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:51:28.621680 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71ad748_a399_48ba_84b5_e9dff218d96f.slice/crio-8394f3ca22948aa564f473f5959af15643033263d5f1e140f4fe53b14cbcecbd WatchSource:0}: Error finding container 8394f3ca22948aa564f473f5959af15643033263d5f1e140f4fe53b14cbcecbd: Status 404 returned error can't find the container with id 8394f3ca22948aa564f473f5959af15643033263d5f1e140f4fe53b14cbcecbd Nov 03 22:51:29.205997 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:29.205957 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" event={"ID":"1eb52cd6-fc70-46e4-8e97-a500d0496c09","Type":"ContainerStarted","Data":"9af89b580a3f3b860569541b0ee4ea9ee885ff36ce49894386c0e47693bc0f05"} Nov 03 22:51:29.206205 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:29.206029 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" event={"ID":"1eb52cd6-fc70-46e4-8e97-a500d0496c09","Type":"ContainerStarted","Data":"b90410d782022e27117f6c1a34fdf7410bc55f038a75d69fa47ccc7902515cc8"} Nov 03 22:51:29.207657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:29.207628 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" event={"ID":"e71ad748-a399-48ba-84b5-e9dff218d96f","Type":"ContainerStarted","Data":"1ab91ea7b14ca9a99bd18dca0e5fb92ebee60de11e7d167b8871a72ad7e27f2d"} Nov 03 22:51:29.207791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:29.207675 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" event={"ID":"e71ad748-a399-48ba-84b5-e9dff218d96f","Type":"ContainerStarted","Data":"8394f3ca22948aa564f473f5959af15643033263d5f1e140f4fe53b14cbcecbd"} Nov 03 22:51:29.245703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:29.245607 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" podStartSLOduration=2.245593826 podStartE2EDuration="2.245593826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:29.241778109 +0000 UTC m=+2315.645719093" watchObservedRunningTime="2026-11-03 22:51:29.245593826 +0000 UTC m=+2315.649534800" Nov 03 22:51:29.285616 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:29.285560 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" podStartSLOduration=2.285547293 podStartE2EDuration="2.285547293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:29.280307889 +0000 UTC m=+2315.684248856" watchObservedRunningTime="2026-11-03 22:51:29.285547293 +0000 UTC m=+2315.689488268" Nov 03 22:51:30.217485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:30.217445 4903 generic.go:356] "Generic (PLEG): container finished" podID="1eb52cd6-fc70-46e4-8e97-a500d0496c09" containerID="9af89b580a3f3b860569541b0ee4ea9ee885ff36ce49894386c0e47693bc0f05" exitCode=0 Nov 03 22:51:30.217679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:30.217525 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" event={"ID":"1eb52cd6-fc70-46e4-8e97-a500d0496c09","Type":"ContainerDied","Data":"9af89b580a3f3b860569541b0ee4ea9ee885ff36ce49894386c0e47693bc0f05"} Nov 03 22:51:30.219171 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:30.219147 4903 generic.go:356] "Generic (PLEG): container finished" podID="e71ad748-a399-48ba-84b5-e9dff218d96f" containerID="1ab91ea7b14ca9a99bd18dca0e5fb92ebee60de11e7d167b8871a72ad7e27f2d" exitCode=0 Nov 03 22:51:30.219323 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:30.219227 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" event={"ID":"e71ad748-a399-48ba-84b5-e9dff218d96f","Type":"ContainerDied","Data":"1ab91ea7b14ca9a99bd18dca0e5fb92ebee60de11e7d167b8871a72ad7e27f2d"} Nov 03 22:51:31.407742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.407715 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:31.451750 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.451724 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:31.517080 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.516995 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb52cd6-fc70-46e4-8e97-a500d0496c09-kube-api-access-lgktp\" (UniqueName: \"kubernetes.io/projected/1eb52cd6-fc70-46e4-8e97-a500d0496c09-kube-api-access-lgktp\") pod \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " Nov 03 22:51:31.517238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.517131 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-xks-ca\") pod \"e71ad748-a399-48ba-84b5-e9dff218d96f\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " Nov 03 22:51:31.517238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.517172 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71ad748-a399-48ba-84b5-e9dff218d96f-kube-api-access-s8xsj\" (UniqueName: \"kubernetes.io/projected/e71ad748-a399-48ba-84b5-e9dff218d96f-kube-api-access-s8xsj\") pod \"e71ad748-a399-48ba-84b5-e9dff218d96f\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " Nov 03 22:51:31.517238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.517223 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-xks-ca\") pod \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " Nov 03 22:51:31.517465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.517319 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-openshift-ca\") pod \"e71ad748-a399-48ba-84b5-e9dff218d96f\" (UID: \"e71ad748-a399-48ba-84b5-e9dff218d96f\") " Nov 03 22:51:31.517465 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.517357 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-openshift-ca\") pod \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\" (UID: \"1eb52cd6-fc70-46e4-8e97-a500d0496c09\") " Nov 03 22:51:31.517863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.517830 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-openshift-ca" pod "e71ad748-a399-48ba-84b5-e9dff218d96f" (UID: "e71ad748-a399-48ba-84b5-e9dff218d96f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:31.517960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.517931 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-openshift-ca" pod "1eb52cd6-fc70-46e4-8e97-a500d0496c09" (UID: "1eb52cd6-fc70-46e4-8e97-a500d0496c09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:51:31.518110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.518087 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:31.518171 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.518117 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:31.523333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.519902 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-xks-ca" pod "1eb52cd6-fc70-46e4-8e97-a500d0496c09" (UID: "1eb52cd6-fc70-46e4-8e97-a500d0496c09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:31.523333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.519979 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71ad748-a399-48ba-84b5-e9dff218d96f-kube-api-access-s8xsj" pod "e71ad748-a399-48ba-84b5-e9dff218d96f" (UID: "e71ad748-a399-48ba-84b5-e9dff218d96f"). InnerVolumeSpecName "kube-api-access-s8xsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:31.523333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.520099 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb52cd6-fc70-46e4-8e97-a500d0496c09-kube-api-access-lgktp" pod "1eb52cd6-fc70-46e4-8e97-a500d0496c09" (UID: "1eb52cd6-fc70-46e4-8e97-a500d0496c09"). InnerVolumeSpecName "kube-api-access-lgktp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:31.523333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.520123 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-xks-ca" pod "e71ad748-a399-48ba-84b5-e9dff218d96f" (UID: "e71ad748-a399-48ba-84b5-e9dff218d96f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:31.620041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.620002 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgktp\" (UniqueName: \"kubernetes.io/projected/1eb52cd6-fc70-46e4-8e97-a500d0496c09-kube-api-access-lgktp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:31.620041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.620035 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71ad748-a399-48ba-84b5-e9dff218d96f-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:31.620041 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.620046 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8xsj\" (UniqueName: \"kubernetes.io/projected/e71ad748-a399-48ba-84b5-e9dff218d96f-kube-api-access-s8xsj\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:31.620338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:31.620055 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb52cd6-fc70-46e4-8e97-a500d0496c09-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:32.236366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:32.236337 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" Nov 03 22:51:32.236584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:32.236336 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q" event={"ID":"1eb52cd6-fc70-46e4-8e97-a500d0496c09","Type":"ContainerDied","Data":"b90410d782022e27117f6c1a34fdf7410bc55f038a75d69fa47ccc7902515cc8"} Nov 03 22:51:32.236584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:32.236466 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b90410d782022e27117f6c1a34fdf7410bc55f038a75d69fa47ccc7902515cc8" Nov 03 22:51:32.237788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:32.237769 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" Nov 03 22:51:32.237788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:32.237778 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6" event={"ID":"e71ad748-a399-48ba-84b5-e9dff218d96f","Type":"ContainerDied","Data":"8394f3ca22948aa564f473f5959af15643033263d5f1e140f4fe53b14cbcecbd"} Nov 03 22:51:32.238012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:32.237812 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8394f3ca22948aa564f473f5959af15643033263d5f1e140f4fe53b14cbcecbd" Nov 03 22:51:37.987749 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:37.987715 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5"] Nov 03 22:51:38.833864 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:38.833830 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth"] Nov 03 22:51:40.229933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.229898 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f"] Nov 03 22:51:40.230716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.230699 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e71ad748-a399-48ba-84b5-e9dff218d96f" containerName="revoke-keys" Nov 03 22:51:40.230777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.230717 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71ad748-a399-48ba-84b5-e9dff218d96f" containerName="revoke-keys" Nov 03 22:51:40.230777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.230739 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1eb52cd6-fc70-46e4-8e97-a500d0496c09" containerName="revoke-keys" Nov 03 22:51:40.230777 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.230744 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb52cd6-fc70-46e4-8e97-a500d0496c09" containerName="revoke-keys" Nov 03 22:51:40.230882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.230873 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71ad748-a399-48ba-84b5-e9dff218d96f" containerName="revoke-keys" Nov 03 22:51:40.230921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.230885 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb52cd6-fc70-46e4-8e97-a500d0496c09" containerName="revoke-keys" Nov 03 22:51:40.235875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.235857 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:40.252499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.252471 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f"] Nov 03 22:51:40.257505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.257483 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-9b841d\"" Nov 03 22:51:40.411780 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.411736 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfbc6\" (UniqueName: \"kubernetes.io/projected/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-kube-api-access-sfbc6\") pod \"payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f\" (UID: \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:40.412265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.412241 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f\" (UID: \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:40.513652 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.513564 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfbc6\" (UniqueName: \"kubernetes.io/projected/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-kube-api-access-sfbc6\") pod \"payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f\" (UID: \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:40.513839 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.513733 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f\" (UID: \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:40.514517 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.514498 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f\" (UID: \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:40.535278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.535245 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfbc6\" (UniqueName: \"kubernetes.io/projected/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-kube-api-access-sfbc6\") pod \"payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f\" (UID: \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:40.551922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.551884 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:40.859310 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:51:40.859260 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe7a73b_4510_41ab_accb_b1e4b31cc4ab.slice/crio-015ff5e4716a2d28f32d3da7f62a8a6822c31632749c181b1aaff97aaa28c45b WatchSource:0}: Error finding container 015ff5e4716a2d28f32d3da7f62a8a6822c31632749c181b1aaff97aaa28c45b: Status 404 returned error can't find the container with id 015ff5e4716a2d28f32d3da7f62a8a6822c31632749c181b1aaff97aaa28c45b Nov 03 22:51:40.859502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:40.859479 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f"] Nov 03 22:51:41.316485 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:41.316443 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" event={"ID":"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab","Type":"ContainerStarted","Data":"67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5"} Nov 03 22:51:41.316886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:41.316513 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:41.316886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:41.316539 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" event={"ID":"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab","Type":"ContainerStarted","Data":"015ff5e4716a2d28f32d3da7f62a8a6822c31632749c181b1aaff97aaa28c45b"} Nov 03 22:51:41.355382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:41.355320 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" podStartSLOduration=1.355287881 podStartE2EDuration="1.355287881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:41.352739715 +0000 UTC m=+2327.756680692" watchObservedRunningTime="2026-11-03 22:51:41.355287881 +0000 UTC m=+2327.759228855" Nov 03 22:51:42.244101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.244061 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" podUID="481dd564-4ea1-48d3-8266-56fa2fb76b75" containerName="main" containerID="cri-o://eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9" gracePeriod=45 Nov 03 22:51:42.550109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.550079 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:51:42.646790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.646756 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481dd564-4ea1-48d3-8266-56fa2fb76b75-tls-certs\" (UniqueName: \"kubernetes.io/secret/481dd564-4ea1-48d3-8266-56fa2fb76b75-tls-certs\") pod \"481dd564-4ea1-48d3-8266-56fa2fb76b75\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " Nov 03 22:51:42.646988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.646796 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-model-cache\") pod \"481dd564-4ea1-48d3-8266-56fa2fb76b75\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " Nov 03 22:51:42.646988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.646847 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-home\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-home\") pod \"481dd564-4ea1-48d3-8266-56fa2fb76b75\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " Nov 03 22:51:42.646988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.646882 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-dshm\") pod \"481dd564-4ea1-48d3-8266-56fa2fb76b75\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " Nov 03 22:51:42.646988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.646988 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-tmp-dir\") pod \"481dd564-4ea1-48d3-8266-56fa2fb76b75\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " Nov 03 22:51:42.647239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.647098 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-home" pod "481dd564-4ea1-48d3-8266-56fa2fb76b75" (UID: "481dd564-4ea1-48d3-8266-56fa2fb76b75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:42.647239 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.647113 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-model-cache" pod "481dd564-4ea1-48d3-8266-56fa2fb76b75" (UID: "481dd564-4ea1-48d3-8266-56fa2fb76b75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:42.647352 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.647245 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481dd564-4ea1-48d3-8266-56fa2fb76b75-kube-api-access-fcgsk\" (UniqueName: \"kubernetes.io/projected/481dd564-4ea1-48d3-8266-56fa2fb76b75-kube-api-access-fcgsk\") pod \"481dd564-4ea1-48d3-8266-56fa2fb76b75\" (UID: \"481dd564-4ea1-48d3-8266-56fa2fb76b75\") " Nov 03 22:51:42.647352 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.647253 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-tmp-dir" pod "481dd564-4ea1-48d3-8266-56fa2fb76b75" (UID: "481dd564-4ea1-48d3-8266-56fa2fb76b75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:42.648020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.647993 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:42.648148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.648024 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:42.648148 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.648038 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:42.649309 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.649273 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-dshm" pod "481dd564-4ea1-48d3-8266-56fa2fb76b75" (UID: "481dd564-4ea1-48d3-8266-56fa2fb76b75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:51:42.649406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.649280 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481dd564-4ea1-48d3-8266-56fa2fb76b75-tls-certs" pod "481dd564-4ea1-48d3-8266-56fa2fb76b75" (UID: "481dd564-4ea1-48d3-8266-56fa2fb76b75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:51:42.649406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.649365 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481dd564-4ea1-48d3-8266-56fa2fb76b75-kube-api-access-fcgsk" pod "481dd564-4ea1-48d3-8266-56fa2fb76b75" (UID: "481dd564-4ea1-48d3-8266-56fa2fb76b75"). InnerVolumeSpecName "kube-api-access-fcgsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:51:42.749836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.749806 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcgsk\" (UniqueName: \"kubernetes.io/projected/481dd564-4ea1-48d3-8266-56fa2fb76b75-kube-api-access-fcgsk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:42.749836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.749834 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481dd564-4ea1-48d3-8266-56fa2fb76b75-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:42.749836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.749844 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481dd564-4ea1-48d3-8266-56fa2fb76b75-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:51:42.988644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:42.988602 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" podUID="b8b704aa-4636-40ca-819e-d7a18e55380a" containerName="payload-pre-processing" containerID="cri-o://f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88" gracePeriod=25 Nov 03 22:51:43.333980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.333882 4903 generic.go:356] "Generic (PLEG): container finished" podID="481dd564-4ea1-48d3-8266-56fa2fb76b75" containerID="eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9" exitCode=0 Nov 03 22:51:43.333980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.333932 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" Nov 03 22:51:43.334221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.333972 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" event={"ID":"481dd564-4ea1-48d3-8266-56fa2fb76b75","Type":"ContainerDied","Data":"eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9"} Nov 03 22:51:43.334221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.334028 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx" event={"ID":"481dd564-4ea1-48d3-8266-56fa2fb76b75","Type":"ContainerDied","Data":"b279affbc2a80d0d895d3f0d6318db683579ce22f60c87457be105f5a72c198e"} Nov 03 22:51:43.334221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.334052 4903 scope.go:122] "RemoveContainer" containerID="eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9" Nov 03 22:51:43.347360 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.347334 4903 scope.go:122] "RemoveContainer" containerID="eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9" Nov 03 22:51:43.347677 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:51:43.347651 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9\": container with ID starting with eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9 not found: ID does not exist" containerID="eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9" Nov 03 22:51:43.347742 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.347688 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9"} err="failed to get container status \"eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9\": rpc error: code = NotFound desc = could not find container \"eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9\": container with ID starting with eaecd366eed6fa0b2ea06ae83070cd4e942ca87cbbabc4b2de391269bb028cd9 not found: ID does not exist" Nov 03 22:51:43.383751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.383716 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx"] Nov 03 22:51:43.393925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.393900 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ccdb60/test-model-ccdb60-kserve-655f55459-dvvqx"] Nov 03 22:51:43.735439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.735399 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481dd564-4ea1-48d3-8266-56fa2fb76b75" path="/var/lib/kubelet/pods/481dd564-4ea1-48d3-8266-56fa2fb76b75/volumes" Nov 03 22:51:43.834605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:43.834566 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" podUID="495cc6a9-b56d-41a2-82d4-3bb5998a84d1" containerName="payload-pre-processing" containerID="cri-o://bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299" gracePeriod=25 Nov 03 22:51:44.177135 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:44.177097 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" podUID="495cc6a9-b56d-41a2-82d4-3bb5998a84d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:46.987405 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:46.987369 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" podUID="b8b704aa-4636-40ca-819e-d7a18e55380a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:48.388192 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:51:48.388152 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:51:49.713653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:49.713612 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q"] Nov 03 22:51:49.722190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:49.722155 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7940ea-26j8q"] Nov 03 22:51:49.737260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:49.737223 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb52cd6-fc70-46e4-8e97-a500d0496c09" path="/var/lib/kubelet/pods/1eb52cd6-fc70-46e4-8e97-a500d0496c09/volumes" Nov 03 22:51:52.328555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:52.328519 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 22:51:54.176849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:54.176809 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" podUID="495cc6a9-b56d-41a2-82d4-3bb5998a84d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:56.902064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:56.902030 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq"] Nov 03 22:51:56.902566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:56.902278 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerName="istio-proxy" containerID="cri-o://0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545" gracePeriod=30 Nov 03 22:51:56.987340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:56.987272 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" podUID="b8b704aa-4636-40ca-819e-d7a18e55380a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:51:57.506344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.506303 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp"] Nov 03 22:51:57.507155 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.507137 4903 cpu_manager.go:414] "containerMap: removing container" podUID="481dd564-4ea1-48d3-8266-56fa2fb76b75" containerName="main" Nov 03 22:51:57.507219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.507156 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481dd564-4ea1-48d3-8266-56fa2fb76b75" containerName="main" Nov 03 22:51:57.507288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.507278 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="481dd564-4ea1-48d3-8266-56fa2fb76b75" containerName="main" Nov 03 22:51:57.511192 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.511174 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.528591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.528567 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp"] Nov 03 22:51:57.533119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.533094 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:51:57.533269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.533146 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:51:57.533435 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.533416 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:51:57.533502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.533494 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:51:57.575027 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.574990 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.575257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.575110 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.575394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.575273 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xsm\" (UniqueName: \"kubernetes.io/projected/66d95716-0400-425f-bbdc-9e5394e1952d-kube-api-access-r4xsm\") pod \"maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.676818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.676778 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.677013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.676870 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.677013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.676983 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xsm\" (UniqueName: \"kubernetes.io/projected/66d95716-0400-425f-bbdc-9e5394e1952d-kube-api-access-r4xsm\") pod \"maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.677781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.677754 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.679368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.679345 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.698920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.698886 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xsm\" (UniqueName: \"kubernetes.io/projected/66d95716-0400-425f-bbdc-9e5394e1952d-kube-api-access-r4xsm\") pod \"maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:57.826453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:57.826374 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:51:58.030069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:58.030036 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88"] Nov 03 22:51:58.034079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:58.034053 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9680aa83-sjk88"] Nov 03 22:51:58.142730 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:51:58.142700 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d95716_0400_425f_bbdc_9e5394e1952d.slice/crio-6c82756ac7b1ad14fae3c0afd3b4d62a4e2ed0bffc4403276c2964c4877f3371 WatchSource:0}: Error finding container 6c82756ac7b1ad14fae3c0afd3b4d62a4e2ed0bffc4403276c2964c4877f3371: Status 404 returned error can't find the container with id 6c82756ac7b1ad14fae3c0afd3b4d62a4e2ed0bffc4403276c2964c4877f3371 Nov 03 22:51:58.143921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:58.143900 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp"] Nov 03 22:51:58.479486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:58.479448 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" event={"ID":"66d95716-0400-425f-bbdc-9e5394e1952d","Type":"ContainerStarted","Data":"810f64c76476af2dd0d608da191c538cb0733c9082d29b18268a5cf6bf730488"} Nov 03 22:51:58.479626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:58.479510 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" event={"ID":"66d95716-0400-425f-bbdc-9e5394e1952d","Type":"ContainerStarted","Data":"6c82756ac7b1ad14fae3c0afd3b4d62a4e2ed0bffc4403276c2964c4877f3371"} Nov 03 22:51:58.516283 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:58.516239 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" podStartSLOduration=1.516225767 podStartE2EDuration="1.516225767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:51:58.51573782 +0000 UTC m=+2344.919678797" watchObservedRunningTime="2026-11-03 22:51:58.516225767 +0000 UTC m=+2344.920166742" Nov 03 22:51:58.732419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:58.732333 4903 scope.go:122] "RemoveContainer" containerID="c6d78eb1deb5249ac5bb14c5bd582bbce4d56aaf50357c9551597ddbc93e7d56" Nov 03 22:51:59.491187 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:59.491141 4903 generic.go:356] "Generic (PLEG): container finished" podID="66d95716-0400-425f-bbdc-9e5394e1952d" containerID="810f64c76476af2dd0d608da191c538cb0733c9082d29b18268a5cf6bf730488" exitCode=0 Nov 03 22:51:59.491725 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:59.491281 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" event={"ID":"66d95716-0400-425f-bbdc-9e5394e1952d","Type":"ContainerDied","Data":"810f64c76476af2dd0d608da191c538cb0733c9082d29b18268a5cf6bf730488"} Nov 03 22:51:59.736331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:51:59.736283 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2466f24f-1e0c-49d2-ba9e-dba2e8e3f967" path="/var/lib/kubelet/pods/2466f24f-1e0c-49d2-ba9e-dba2e8e3f967/volumes" Nov 03 22:52:00.524381 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:00.524278 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:52:00.807911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:00.807873 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:52:00.931885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:00.931846 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d95716-0400-425f-bbdc-9e5394e1952d-kube-api-access-r4xsm\" (UniqueName: \"kubernetes.io/projected/66d95716-0400-425f-bbdc-9e5394e1952d-kube-api-access-r4xsm\") pod \"66d95716-0400-425f-bbdc-9e5394e1952d\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " Nov 03 22:52:00.932070 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:00.931989 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-xks-ca\") pod \"66d95716-0400-425f-bbdc-9e5394e1952d\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " Nov 03 22:52:00.932180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:00.932163 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-openshift-ca\") pod \"66d95716-0400-425f-bbdc-9e5394e1952d\" (UID: \"66d95716-0400-425f-bbdc-9e5394e1952d\") " Nov 03 22:52:00.932480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:00.932453 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-openshift-ca" pod "66d95716-0400-425f-bbdc-9e5394e1952d" (UID: "66d95716-0400-425f-bbdc-9e5394e1952d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:00.932824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:00.932800 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:00.934124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:00.934098 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-xks-ca" pod "66d95716-0400-425f-bbdc-9e5394e1952d" (UID: "66d95716-0400-425f-bbdc-9e5394e1952d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:00.934214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:00.934109 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d95716-0400-425f-bbdc-9e5394e1952d-kube-api-access-r4xsm" pod "66d95716-0400-425f-bbdc-9e5394e1952d" (UID: "66d95716-0400-425f-bbdc-9e5394e1952d"). InnerVolumeSpecName "kube-api-access-r4xsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:01.033823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:01.033737 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4xsm\" (UniqueName: \"kubernetes.io/projected/66d95716-0400-425f-bbdc-9e5394e1952d-kube-api-access-r4xsm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:01.033823 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:01.033769 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d95716-0400-425f-bbdc-9e5394e1952d-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:01.520090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:01.520046 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" event={"ID":"66d95716-0400-425f-bbdc-9e5394e1952d","Type":"ContainerDied","Data":"6c82756ac7b1ad14fae3c0afd3b4d62a4e2ed0bffc4403276c2964c4877f3371"} Nov 03 22:52:01.520330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:01.520143 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c82756ac7b1ad14fae3c0afd3b4d62a4e2ed0bffc4403276c2964c4877f3371" Nov 03 22:52:01.520330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:01.520213 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp" Nov 03 22:52:02.332284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.332248 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:52:02.468883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.468851 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-envoy\") pod \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " Nov 03 22:52:02.469163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.468917 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-certs\") pod \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " Nov 03 22:52:02.469163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469057 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-kube-api-access-6mlg6\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-kube-api-access-6mlg6\") pod \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " Nov 03 22:52:02.469163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469129 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istiod-ca-cert\") pod \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " Nov 03 22:52:02.469407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469206 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-certs" pod "a619613c-1ab5-4af3-b687-dd4a9a3f6abb" (UID: "a619613c-1ab5-4af3-b687-dd4a9a3f6abb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.469407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469214 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-socket\") pod \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " Nov 03 22:52:02.469407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469331 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-credential-socket\") pod \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " Nov 03 22:52:02.469407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469379 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-podinfo\") pod \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " Nov 03 22:52:02.469633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469474 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-socket" pod "a619613c-1ab5-4af3-b687-dd4a9a3f6abb" (UID: "a619613c-1ab5-4af3-b687-dd4a9a3f6abb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.469633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469497 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-data\") pod \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " Nov 03 22:52:02.469633 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469531 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-token\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-token\") pod \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\" (UID: \"a619613c-1ab5-4af3-b687-dd4a9a3f6abb\") " Nov 03 22:52:02.469806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469633 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-credential-socket" pod "a619613c-1ab5-4af3-b687-dd4a9a3f6abb" (UID: "a619613c-1ab5-4af3-b687-dd4a9a3f6abb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.469806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.469665 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istiod-ca-cert" pod "a619613c-1ab5-4af3-b687-dd4a9a3f6abb" (UID: "a619613c-1ab5-4af3-b687-dd4a9a3f6abb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:02.470092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.470064 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-data" pod "a619613c-1ab5-4af3-b687-dd4a9a3f6abb" (UID: "a619613c-1ab5-4af3-b687-dd4a9a3f6abb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.470363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.470335 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.470363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.470361 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.470540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.470375 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.470540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.470391 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.470540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.470405 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.471733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.471708 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-podinfo" pod "a619613c-1ab5-4af3-b687-dd4a9a3f6abb" (UID: "a619613c-1ab5-4af3-b687-dd4a9a3f6abb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:52:02.471846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.471726 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-kube-api-access-6mlg6" pod "a619613c-1ab5-4af3-b687-dd4a9a3f6abb" (UID: "a619613c-1ab5-4af3-b687-dd4a9a3f6abb"). InnerVolumeSpecName "kube-api-access-6mlg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:02.471846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.471798 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-envoy" pod "a619613c-1ab5-4af3-b687-dd4a9a3f6abb" (UID: "a619613c-1ab5-4af3-b687-dd4a9a3f6abb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:02.472030 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.472012 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-token" pod "a619613c-1ab5-4af3-b687-dd4a9a3f6abb" (UID: "a619613c-1ab5-4af3-b687-dd4a9a3f6abb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:02.530484 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.530440 4903 generic.go:356] "Generic (PLEG): container finished" podID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerID="0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545" exitCode=0 Nov 03 22:52:02.530697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.530494 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" event={"ID":"a619613c-1ab5-4af3-b687-dd4a9a3f6abb","Type":"ContainerDied","Data":"0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545"} Nov 03 22:52:02.530697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.530543 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" event={"ID":"a619613c-1ab5-4af3-b687-dd4a9a3f6abb","Type":"ContainerDied","Data":"e261abf5dec49645f2b33a3be76b204137169a63b46d8f6f9377cb60741b6ff7"} Nov 03 22:52:02.530697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.530565 4903 scope.go:122] "RemoveContainer" containerID="0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545" Nov 03 22:52:02.530697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.530498 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq" Nov 03 22:52:02.543362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.543258 4903 scope.go:122] "RemoveContainer" containerID="0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545" Nov 03 22:52:02.543682 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:52:02.543658 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545\": container with ID starting with 0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545 not found: ID does not exist" containerID="0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545" Nov 03 22:52:02.543734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.543690 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545"} err="failed to get container status \"0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545\": rpc error: code = NotFound desc = could not find container \"0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545\": container with ID starting with 0712605f3062caf1450f443431de51b201e73bd96701ba985bf0a9583dfb0545 not found: ID does not exist" Nov 03 22:52:02.571915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.571878 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.571915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.571917 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mlg6\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-kube-api-access-6mlg6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.572124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.571934 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.572124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.571949 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a619613c-1ab5-4af3-b687-dd4a9a3f6abb-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:02.585956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.585921 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq"] Nov 03 22:52:02.592866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:02.592838 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7940ea-openshift-default-75b99d856c-n9xtq"] Nov 03 22:52:03.317950 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.317924 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:52:03.495010 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.494975 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b704aa-4636-40ca-819e-d7a18e55380a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b704aa-4636-40ca-819e-d7a18e55380a-plugins-config-volume\") pod \"b8b704aa-4636-40ca-819e-d7a18e55380a\" (UID: \"b8b704aa-4636-40ca-819e-d7a18e55380a\") " Nov 03 22:52:03.495442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.495023 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b704aa-4636-40ca-819e-d7a18e55380a-kube-api-access-55rz2\" (UniqueName: \"kubernetes.io/projected/b8b704aa-4636-40ca-819e-d7a18e55380a-kube-api-access-55rz2\") pod \"b8b704aa-4636-40ca-819e-d7a18e55380a\" (UID: \"b8b704aa-4636-40ca-819e-d7a18e55380a\") " Nov 03 22:52:03.495497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.495455 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b704aa-4636-40ca-819e-d7a18e55380a-plugins-config-volume" pod "b8b704aa-4636-40ca-819e-d7a18e55380a" (UID: "b8b704aa-4636-40ca-819e-d7a18e55380a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:03.495663 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.495648 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8b704aa-4636-40ca-819e-d7a18e55380a-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:03.497158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.497128 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b704aa-4636-40ca-819e-d7a18e55380a-kube-api-access-55rz2" pod "b8b704aa-4636-40ca-819e-d7a18e55380a" (UID: "b8b704aa-4636-40ca-819e-d7a18e55380a"). InnerVolumeSpecName "kube-api-access-55rz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:03.540197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.540114 4903 generic.go:356] "Generic (PLEG): container finished" podID="b8b704aa-4636-40ca-819e-d7a18e55380a" containerID="f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88" exitCode=0 Nov 03 22:52:03.540197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.540161 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" Nov 03 22:52:03.540440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.540191 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" event={"ID":"b8b704aa-4636-40ca-819e-d7a18e55380a","Type":"ContainerDied","Data":"f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88"} Nov 03 22:52:03.540440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.540255 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5" event={"ID":"b8b704aa-4636-40ca-819e-d7a18e55380a","Type":"ContainerDied","Data":"d06f4fe06dc6d4a51f12f2dfb56b1f2a330d2cabb66bd147639efc5eccba4818"} Nov 03 22:52:03.540440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.540286 4903 scope.go:122] "RemoveContainer" containerID="f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88" Nov 03 22:52:03.552891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.552876 4903 scope.go:122] "RemoveContainer" containerID="f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88" Nov 03 22:52:03.553158 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:52:03.553140 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88\": container with ID starting with f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88 not found: ID does not exist" containerID="f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88" Nov 03 22:52:03.553214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.553166 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88"} err="failed to get container status \"f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88\": rpc error: code = NotFound desc = could not find container \"f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88\": container with ID starting with f100a349d83830979317cd692612f6b98dea8e80de387570b379cd30c6890a88 not found: ID does not exist" Nov 03 22:52:03.588714 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.588687 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5"] Nov 03 22:52:03.592662 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.592638 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7940ea-6c48467979-8rjd5"] Nov 03 22:52:03.597168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.597143 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55rz2\" (UniqueName: \"kubernetes.io/projected/b8b704aa-4636-40ca-819e-d7a18e55380a-kube-api-access-55rz2\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:03.750273 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.750236 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" path="/var/lib/kubelet/pods/a619613c-1ab5-4af3-b687-dd4a9a3f6abb/volumes" Nov 03 22:52:03.750886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:03.750867 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b704aa-4636-40ca-819e-d7a18e55380a" path="/var/lib/kubelet/pods/b8b704aa-4636-40ca-819e-d7a18e55380a/volumes" Nov 03 22:52:04.122118 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.122092 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:52:04.305371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.305333 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-kube-api-access-8mz4x\" (UniqueName: \"kubernetes.io/projected/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-kube-api-access-8mz4x\") pod \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\" (UID: \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\") " Nov 03 22:52:04.305556 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.305535 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-plugins-config-volume\") pod \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\" (UID: \"495cc6a9-b56d-41a2-82d4-3bb5998a84d1\") " Nov 03 22:52:04.305900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.305873 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-plugins-config-volume" pod "495cc6a9-b56d-41a2-82d4-3bb5998a84d1" (UID: "495cc6a9-b56d-41a2-82d4-3bb5998a84d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:04.307606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.307581 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-kube-api-access-8mz4x" pod "495cc6a9-b56d-41a2-82d4-3bb5998a84d1" (UID: "495cc6a9-b56d-41a2-82d4-3bb5998a84d1"). InnerVolumeSpecName "kube-api-access-8mz4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:04.407375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.407340 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:04.407375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.407369 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mz4x\" (UniqueName: \"kubernetes.io/projected/495cc6a9-b56d-41a2-82d4-3bb5998a84d1-kube-api-access-8mz4x\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:04.555680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.555639 4903 generic.go:356] "Generic (PLEG): container finished" podID="495cc6a9-b56d-41a2-82d4-3bb5998a84d1" containerID="bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299" exitCode=0 Nov 03 22:52:04.556267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.555691 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" Nov 03 22:52:04.556267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.555724 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" event={"ID":"495cc6a9-b56d-41a2-82d4-3bb5998a84d1","Type":"ContainerDied","Data":"bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299"} Nov 03 22:52:04.556267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.555797 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth" event={"ID":"495cc6a9-b56d-41a2-82d4-3bb5998a84d1","Type":"ContainerDied","Data":"324194690a485a5bece5a2aef1f50fcd8ac50a83f0ea45391e7dc4fd4731f7f6"} Nov 03 22:52:04.556267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.555830 4903 scope.go:122] "RemoveContainer" containerID="bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299" Nov 03 22:52:04.571563 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.571534 4903 scope.go:122] "RemoveContainer" containerID="bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299" Nov 03 22:52:04.571881 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:52:04.571860 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299\": container with ID starting with bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299 not found: ID does not exist" containerID="bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299" Nov 03 22:52:04.571970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.571894 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299"} err="failed to get container status \"bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299\": rpc error: code = NotFound desc = could not find container \"bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299\": container with ID starting with bddb2081520038d25009ab2f0d185a32ec535553ac60557eb1f2df6cb87ae299 not found: ID does not exist" Nov 03 22:52:04.604163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.604127 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth"] Nov 03 22:52:04.608436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:04.608411 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b9b5-76b76f77c9-g6sth"] Nov 03 22:52:05.737276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:05.737237 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495cc6a9-b56d-41a2-82d4-3bb5998a84d1" path="/var/lib/kubelet/pods/495cc6a9-b56d-41a2-82d4-3bb5998a84d1/volumes" Nov 03 22:52:07.953731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:07.953701 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs"] Nov 03 22:52:09.029017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:09.028982 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm"] Nov 03 22:52:09.039157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:09.039130 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1802c6fa-lvtjm"] Nov 03 22:52:09.735250 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:09.735215 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416cc0c8-1592-479d-ba47-cc57da5059a1" path="/var/lib/kubelet/pods/416cc0c8-1592-479d-ba47-cc57da5059a1/volumes" Nov 03 22:52:12.954210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:12.954166 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" podUID="ea6fcce1-a565-4bd4-a769-acbeb168d0d3" containerName="payload-processing" containerID="cri-o://0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5" gracePeriod=25 Nov 03 22:52:14.329631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:14.329600 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6"] Nov 03 22:52:14.338862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:14.338825 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b9b5-qrqb6"] Nov 03 22:52:15.737278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:15.737236 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71ad748-a399-48ba-84b5-e9dff218d96f" path="/var/lib/kubelet/pods/e71ad748-a399-48ba-84b5-e9dff218d96f/volumes" Nov 03 22:52:17.237006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:17.236963 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" podUID="ea6fcce1-a565-4bd4-a769-acbeb168d0d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:52:18.388068 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:52:18.388032 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:52:22.621590 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.621546 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp"] Nov 03 22:52:22.622601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622575 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerName="istio-proxy" Nov 03 22:52:22.622601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622601 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerName="istio-proxy" Nov 03 22:52:22.622794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622624 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b8b704aa-4636-40ca-819e-d7a18e55380a" containerName="payload-pre-processing" Nov 03 22:52:22.622794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622633 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b704aa-4636-40ca-819e-d7a18e55380a" containerName="payload-pre-processing" Nov 03 22:52:22.622794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622662 4903 cpu_manager.go:414] "containerMap: removing container" podUID="495cc6a9-b56d-41a2-82d4-3bb5998a84d1" containerName="payload-pre-processing" Nov 03 22:52:22.622794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622667 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495cc6a9-b56d-41a2-82d4-3bb5998a84d1" containerName="payload-pre-processing" Nov 03 22:52:22.622794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622693 4903 cpu_manager.go:414] "containerMap: removing container" podUID="66d95716-0400-425f-bbdc-9e5394e1952d" containerName="revoke-keys" Nov 03 22:52:22.622794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622698 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d95716-0400-425f-bbdc-9e5394e1952d" containerName="revoke-keys" Nov 03 22:52:22.622988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622821 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d95716-0400-425f-bbdc-9e5394e1952d" containerName="revoke-keys" Nov 03 22:52:22.622988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622835 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a619613c-1ab5-4af3-b687-dd4a9a3f6abb" containerName="istio-proxy" Nov 03 22:52:22.622988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622842 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b704aa-4636-40ca-819e-d7a18e55380a" containerName="payload-pre-processing" Nov 03 22:52:22.622988 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.622852 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="495cc6a9-b56d-41a2-82d4-3bb5998a84d1" containerName="payload-pre-processing" Nov 03 22:52:22.626605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.626579 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.644109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.644075 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp"] Nov 03 22:52:22.648652 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.648617 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:22.648652 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.648632 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:52:22.648871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.648661 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:52:22.648871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.648701 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:52:22.810790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.810753 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.810973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.810908 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwjk\" (UniqueName: \"kubernetes.io/projected/f8a0d143-82bc-4866-95fe-1f757a010762-kube-api-access-cfwjk\") pod \"maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.811094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.811070 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.912176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.912142 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.912400 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.912259 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.912492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.912400 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwjk\" (UniqueName: \"kubernetes.io/projected/f8a0d143-82bc-4866-95fe-1f757a010762-kube-api-access-cfwjk\") pod \"maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.913022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.913001 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.914593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.914571 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.934569 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.934544 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwjk\" (UniqueName: \"kubernetes.io/projected/f8a0d143-82bc-4866-95fe-1f757a010762-kube-api-access-cfwjk\") pod \"maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:22.942518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:22.942492 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:23.251251 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:23.251221 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp"] Nov 03 22:52:23.251500 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:52:23.251477 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a0d143_82bc_4866_95fe_1f757a010762.slice/crio-0eb3bab0bc198cc4c2dbf7fd862bdc5afbae2162ad211b5b0d00e1513f5f31a4 WatchSource:0}: Error finding container 0eb3bab0bc198cc4c2dbf7fd862bdc5afbae2162ad211b5b0d00e1513f5f31a4: Status 404 returned error can't find the container with id 0eb3bab0bc198cc4c2dbf7fd862bdc5afbae2162ad211b5b0d00e1513f5f31a4 Nov 03 22:52:23.764728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:23.764691 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" event={"ID":"f8a0d143-82bc-4866-95fe-1f757a010762","Type":"ContainerStarted","Data":"32fac65ea85cd6b73e89f0ab66473a7f4411dee8bf83227fbc3411934b695680"} Nov 03 22:52:23.765126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:23.764749 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" event={"ID":"f8a0d143-82bc-4866-95fe-1f757a010762","Type":"ContainerStarted","Data":"0eb3bab0bc198cc4c2dbf7fd862bdc5afbae2162ad211b5b0d00e1513f5f31a4"} Nov 03 22:52:23.803781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:23.803726 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" podStartSLOduration=1.803712333 podStartE2EDuration="1.803712333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:23.800630752 +0000 UTC m=+2370.204571727" watchObservedRunningTime="2026-11-03 22:52:23.803712333 +0000 UTC m=+2370.207653305" Nov 03 22:52:24.776071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:24.776032 4903 generic.go:356] "Generic (PLEG): container finished" podID="f8a0d143-82bc-4866-95fe-1f757a010762" containerID="32fac65ea85cd6b73e89f0ab66473a7f4411dee8bf83227fbc3411934b695680" exitCode=0 Nov 03 22:52:24.776560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:24.776116 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" event={"ID":"f8a0d143-82bc-4866-95fe-1f757a010762","Type":"ContainerDied","Data":"32fac65ea85cd6b73e89f0ab66473a7f4411dee8bf83227fbc3411934b695680"} Nov 03 22:52:25.978608 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:25.978580 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:26.057703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.057603 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-xks-ca\") pod \"f8a0d143-82bc-4866-95fe-1f757a010762\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " Nov 03 22:52:26.057877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.057842 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-openshift-ca\") pod \"f8a0d143-82bc-4866-95fe-1f757a010762\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " Nov 03 22:52:26.058172 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.058146 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a0d143-82bc-4866-95fe-1f757a010762-kube-api-access-cfwjk\" (UniqueName: \"kubernetes.io/projected/f8a0d143-82bc-4866-95fe-1f757a010762-kube-api-access-cfwjk\") pod \"f8a0d143-82bc-4866-95fe-1f757a010762\" (UID: \"f8a0d143-82bc-4866-95fe-1f757a010762\") " Nov 03 22:52:26.058367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.058219 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-openshift-ca" pod "f8a0d143-82bc-4866-95fe-1f757a010762" (UID: "f8a0d143-82bc-4866-95fe-1f757a010762"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:26.059096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.059068 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:26.060578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.060550 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-xks-ca" pod "f8a0d143-82bc-4866-95fe-1f757a010762" (UID: "f8a0d143-82bc-4866-95fe-1f757a010762"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:26.060689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.060647 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a0d143-82bc-4866-95fe-1f757a010762-kube-api-access-cfwjk" pod "f8a0d143-82bc-4866-95fe-1f757a010762" (UID: "f8a0d143-82bc-4866-95fe-1f757a010762"). InnerVolumeSpecName "kube-api-access-cfwjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:26.160102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.160067 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwjk\" (UniqueName: \"kubernetes.io/projected/f8a0d143-82bc-4866-95fe-1f757a010762-kube-api-access-cfwjk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:26.160102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.160101 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a0d143-82bc-4866-95fe-1f757a010762-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:26.348043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.347956 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf"] Nov 03 22:52:26.348912 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.348892 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f8a0d143-82bc-4866-95fe-1f757a010762" containerName="revoke-keys" Nov 03 22:52:26.348981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.348913 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a0d143-82bc-4866-95fe-1f757a010762" containerName="revoke-keys" Nov 03 22:52:26.349062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.349052 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a0d143-82bc-4866-95fe-1f757a010762" containerName="revoke-keys" Nov 03 22:52:26.352750 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.352732 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.373110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.373078 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf"] Nov 03 22:52:26.464596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.464562 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/9c1ad657-16aa-48be-9d97-da10f06ab551-kube-api-access-58cq6\") pod \"maas-api-revoke-sub-15ed03e7-95tlf\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.464745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.464676 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ed03e7-95tlf\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.464874 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.464858 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ed03e7-95tlf\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.566541 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.566506 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ed03e7-95tlf\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.566730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.566590 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/9c1ad657-16aa-48be-9d97-da10f06ab551-kube-api-access-58cq6\") pod \"maas-api-revoke-sub-15ed03e7-95tlf\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.566730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.566638 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ed03e7-95tlf\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.567354 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.567332 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15ed03e7-95tlf\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.569083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.569058 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15ed03e7-95tlf\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.588608 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.588578 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/9c1ad657-16aa-48be-9d97-da10f06ab551-kube-api-access-58cq6\") pod \"maas-api-revoke-sub-15ed03e7-95tlf\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.670223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.670193 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:26.804466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.804431 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" Nov 03 22:52:26.804613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.804465 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp" event={"ID":"f8a0d143-82bc-4866-95fe-1f757a010762","Type":"ContainerDied","Data":"0eb3bab0bc198cc4c2dbf7fd862bdc5afbae2162ad211b5b0d00e1513f5f31a4"} Nov 03 22:52:26.804613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.804523 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eb3bab0bc198cc4c2dbf7fd862bdc5afbae2162ad211b5b0d00e1513f5f31a4" Nov 03 22:52:26.977612 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:26.977571 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf"] Nov 03 22:52:26.977925 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:52:26.977902 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1ad657_16aa_48be_9d97_da10f06ab551.slice/crio-c231f1d1d3300c3a1a5e8a0ec84469103121317c4f784c23a36885548690d7c5 WatchSource:0}: Error finding container c231f1d1d3300c3a1a5e8a0ec84469103121317c4f784c23a36885548690d7c5: Status 404 returned error can't find the container with id c231f1d1d3300c3a1a5e8a0ec84469103121317c4f784c23a36885548690d7c5 Nov 03 22:52:27.243367 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:27.242371 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" podUID="ea6fcce1-a565-4bd4-a769-acbeb168d0d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:52:27.814394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:27.814280 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" event={"ID":"9c1ad657-16aa-48be-9d97-da10f06ab551","Type":"ContainerStarted","Data":"4528b7404d663ceb0e3bf4055bfdd9b673ef9f909647a9af6f8e39446156c9b9"} Nov 03 22:52:27.814394 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:27.814370 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" event={"ID":"9c1ad657-16aa-48be-9d97-da10f06ab551","Type":"ContainerStarted","Data":"c231f1d1d3300c3a1a5e8a0ec84469103121317c4f784c23a36885548690d7c5"} Nov 03 22:52:27.858382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:27.858275 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" podStartSLOduration=1.858260094 podStartE2EDuration="1.858260094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:52:27.854542325 +0000 UTC m=+2374.258483300" watchObservedRunningTime="2026-11-03 22:52:27.858260094 +0000 UTC m=+2374.262201069" Nov 03 22:52:28.826132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:28.826098 4903 generic.go:356] "Generic (PLEG): container finished" podID="9c1ad657-16aa-48be-9d97-da10f06ab551" containerID="4528b7404d663ceb0e3bf4055bfdd9b673ef9f909647a9af6f8e39446156c9b9" exitCode=0 Nov 03 22:52:28.826683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:28.826183 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" event={"ID":"9c1ad657-16aa-48be-9d97-da10f06ab551","Type":"ContainerDied","Data":"4528b7404d663ceb0e3bf4055bfdd9b673ef9f909647a9af6f8e39446156c9b9"} Nov 03 22:52:30.007008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.006981 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:30.116622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.116551 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1ad657-16aa-48be-9d97-da10f06ab551-kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/9c1ad657-16aa-48be-9d97-da10f06ab551-kube-api-access-58cq6\") pod \"9c1ad657-16aa-48be-9d97-da10f06ab551\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " Nov 03 22:52:30.116870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.116623 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-openshift-ca\") pod \"9c1ad657-16aa-48be-9d97-da10f06ab551\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " Nov 03 22:52:30.116870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.116773 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-xks-ca\") pod \"9c1ad657-16aa-48be-9d97-da10f06ab551\" (UID: \"9c1ad657-16aa-48be-9d97-da10f06ab551\") " Nov 03 22:52:30.117005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.116961 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-openshift-ca" pod "9c1ad657-16aa-48be-9d97-da10f06ab551" (UID: "9c1ad657-16aa-48be-9d97-da10f06ab551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:30.117339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.117317 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:30.118802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.118779 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-xks-ca" pod "9c1ad657-16aa-48be-9d97-da10f06ab551" (UID: "9c1ad657-16aa-48be-9d97-da10f06ab551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:30.118802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.118789 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1ad657-16aa-48be-9d97-da10f06ab551-kube-api-access-58cq6" pod "9c1ad657-16aa-48be-9d97-da10f06ab551" (UID: "9c1ad657-16aa-48be-9d97-da10f06ab551"). InnerVolumeSpecName "kube-api-access-58cq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:30.218833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.218802 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1ad657-16aa-48be-9d97-da10f06ab551-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:30.218833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.218828 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/9c1ad657-16aa-48be-9d97-da10f06ab551-kube-api-access-58cq6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:30.845249 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.845218 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" event={"ID":"9c1ad657-16aa-48be-9d97-da10f06ab551","Type":"ContainerDied","Data":"c231f1d1d3300c3a1a5e8a0ec84469103121317c4f784c23a36885548690d7c5"} Nov 03 22:52:30.845249 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.845234 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf" Nov 03 22:52:30.845491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:30.845262 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c231f1d1d3300c3a1a5e8a0ec84469103121317c4f784c23a36885548690d7c5" Nov 03 22:52:33.035824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.035786 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs"] Nov 03 22:52:33.306695 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.306582 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:52:33.390229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.390191 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-kube-api-access-qhw5z\" (UniqueName: \"kubernetes.io/projected/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-kube-api-access-qhw5z\") pod \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\" (UID: \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\") " Nov 03 22:52:33.390438 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.390285 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-plugins-config-volume\") pod \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\" (UID: \"ea6fcce1-a565-4bd4-a769-acbeb168d0d3\") " Nov 03 22:52:33.390731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.390699 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-plugins-config-volume" pod "ea6fcce1-a565-4bd4-a769-acbeb168d0d3" (UID: "ea6fcce1-a565-4bd4-a769-acbeb168d0d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:33.391050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.391025 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:33.392626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.392597 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-kube-api-access-qhw5z" pod "ea6fcce1-a565-4bd4-a769-acbeb168d0d3" (UID: "ea6fcce1-a565-4bd4-a769-acbeb168d0d3"). InnerVolumeSpecName "kube-api-access-qhw5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:33.492129 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.492095 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhw5z\" (UniqueName: \"kubernetes.io/projected/ea6fcce1-a565-4bd4-a769-acbeb168d0d3-kube-api-access-qhw5z\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:33.870430 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.870398 4903 generic.go:356] "Generic (PLEG): container finished" podID="ea6fcce1-a565-4bd4-a769-acbeb168d0d3" containerID="0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5" exitCode=0 Nov 03 22:52:33.870614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.870446 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" Nov 03 22:52:33.870614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.870490 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" event={"ID":"ea6fcce1-a565-4bd4-a769-acbeb168d0d3","Type":"ContainerDied","Data":"0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5"} Nov 03 22:52:33.870614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.870548 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs" event={"ID":"ea6fcce1-a565-4bd4-a769-acbeb168d0d3","Type":"ContainerDied","Data":"f2fecb2f9fc9cc503fd4b1c2e47e86d0bb5150cff09e7c517b939d2d3816bca1"} Nov 03 22:52:33.870614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.870571 4903 scope.go:122] "RemoveContainer" containerID="0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5" Nov 03 22:52:33.882267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.882241 4903 scope.go:122] "RemoveContainer" containerID="0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5" Nov 03 22:52:33.882576 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:52:33.882550 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5\": container with ID starting with 0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5 not found: ID does not exist" containerID="0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5" Nov 03 22:52:33.882661 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.882590 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5"} err="failed to get container status \"0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5\": rpc error: code = NotFound desc = could not find container \"0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5\": container with ID starting with 0354cca3c7a07ad27bf08e3a0b199b8ba5bf81d48fbf3515eeb8ea670bc6b3b5 not found: ID does not exist" Nov 03 22:52:33.919158 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.919123 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs"] Nov 03 22:52:33.929277 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:33.929253 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ccdb60-7fc6f896b-qrxhs"] Nov 03 22:52:35.740261 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:35.740221 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6fcce1-a565-4bd4-a769-acbeb168d0d3" path="/var/lib/kubelet/pods/ea6fcce1-a565-4bd4-a769-acbeb168d0d3/volumes" Nov 03 22:52:37.365194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.365162 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp"] Nov 03 22:52:37.375119 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.375092 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccdb60-6g6dp"] Nov 03 22:52:37.415320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.415273 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b"] Nov 03 22:52:37.744018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.743983 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d95716-0400-425f-bbdc-9e5394e1952d" path="/var/lib/kubelet/pods/66d95716-0400-425f-bbdc-9e5394e1952d/volumes" Nov 03 22:52:37.968601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.968572 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p"] Nov 03 22:52:37.969315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.969288 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ea6fcce1-a565-4bd4-a769-acbeb168d0d3" containerName="payload-processing" Nov 03 22:52:37.969375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.969316 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6fcce1-a565-4bd4-a769-acbeb168d0d3" containerName="payload-processing" Nov 03 22:52:37.969375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.969345 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9c1ad657-16aa-48be-9d97-da10f06ab551" containerName="revoke-keys" Nov 03 22:52:37.969375 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.969350 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1ad657-16aa-48be-9d97-da10f06ab551" containerName="revoke-keys" Nov 03 22:52:37.969471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.969458 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6fcce1-a565-4bd4-a769-acbeb168d0d3" containerName="payload-processing" Nov 03 22:52:37.969506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.969472 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1ad657-16aa-48be-9d97-da10f06ab551" containerName="revoke-keys" Nov 03 22:52:37.974269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.974254 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:37.990583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.990554 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p"] Nov 03 22:52:37.995925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.995830 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:52:37.995925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.995863 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:52:37.996123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.995923 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:52:37.996123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:37.995940 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:52:38.036955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.036919 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" podUID="bbb7b18e-b8d1-45dd-99db-999de387124c" containerName="payload-processing" containerID="cri-o://2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2" gracePeriod=25 Nov 03 22:52:38.094361 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.094328 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwv26\" (UniqueName: \"kubernetes.io/projected/2d6bf448-b4ff-4082-a31f-c7a9062769e8-kube-api-access-pwv26\") pod \"maas-api-revoke-sub-f2cae1c3-f9m5p\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.094537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.094395 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2cae1c3-f9m5p\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.094537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.094519 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2cae1c3-f9m5p\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.195783 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.195749 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwv26\" (UniqueName: \"kubernetes.io/projected/2d6bf448-b4ff-4082-a31f-c7a9062769e8-kube-api-access-pwv26\") pod \"maas-api-revoke-sub-f2cae1c3-f9m5p\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.195983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.195843 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2cae1c3-f9m5p\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.195983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.195887 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2cae1c3-f9m5p\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.196687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.196657 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2cae1c3-f9m5p\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.198385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.198365 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2cae1c3-f9m5p\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.234995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.234957 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwv26\" (UniqueName: \"kubernetes.io/projected/2d6bf448-b4ff-4082-a31f-c7a9062769e8-kube-api-access-pwv26\") pod \"maas-api-revoke-sub-f2cae1c3-f9m5p\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.291043 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.290965 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:38.345112 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.345082 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:52:38.399685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.399649 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb7b18e-b8d1-45dd-99db-999de387124c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb7b18e-b8d1-45dd-99db-999de387124c-plugins-config-volume\") pod \"bbb7b18e-b8d1-45dd-99db-999de387124c\" (UID: \"bbb7b18e-b8d1-45dd-99db-999de387124c\") " Nov 03 22:52:38.400081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.399710 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb7b18e-b8d1-45dd-99db-999de387124c-kube-api-access-5xkz5\" (UniqueName: \"kubernetes.io/projected/bbb7b18e-b8d1-45dd-99db-999de387124c-kube-api-access-5xkz5\") pod \"bbb7b18e-b8d1-45dd-99db-999de387124c\" (UID: \"bbb7b18e-b8d1-45dd-99db-999de387124c\") " Nov 03 22:52:38.400081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.400030 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb7b18e-b8d1-45dd-99db-999de387124c-plugins-config-volume" pod "bbb7b18e-b8d1-45dd-99db-999de387124c" (UID: "bbb7b18e-b8d1-45dd-99db-999de387124c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:38.400547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.400522 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb7b18e-b8d1-45dd-99db-999de387124c-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:38.402242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.402208 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb7b18e-b8d1-45dd-99db-999de387124c-kube-api-access-5xkz5" pod "bbb7b18e-b8d1-45dd-99db-999de387124c" (UID: "bbb7b18e-b8d1-45dd-99db-999de387124c"). InnerVolumeSpecName "kube-api-access-5xkz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:38.502000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.501966 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xkz5\" (UniqueName: \"kubernetes.io/projected/bbb7b18e-b8d1-45dd-99db-999de387124c-kube-api-access-5xkz5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:38.605710 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:52:38.605681 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6bf448_b4ff_4082_a31f_c7a9062769e8.slice/crio-d2594a51aab6a6b5797e0f63e549d07f1df6deb042fdc7bf2ec5d2dad6929262 WatchSource:0}: Error finding container d2594a51aab6a6b5797e0f63e549d07f1df6deb042fdc7bf2ec5d2dad6929262: Status 404 returned error can't find the container with id d2594a51aab6a6b5797e0f63e549d07f1df6deb042fdc7bf2ec5d2dad6929262 Nov 03 22:52:38.607097 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.607074 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p"] Nov 03 22:52:38.928657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.928626 4903 generic.go:356] "Generic (PLEG): container finished" podID="bbb7b18e-b8d1-45dd-99db-999de387124c" containerID="2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2" exitCode=0 Nov 03 22:52:38.928844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.928671 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" Nov 03 22:52:38.928844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.928712 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" event={"ID":"bbb7b18e-b8d1-45dd-99db-999de387124c","Type":"ContainerDied","Data":"2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2"} Nov 03 22:52:38.928844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.928769 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs" event={"ID":"bbb7b18e-b8d1-45dd-99db-999de387124c","Type":"ContainerDied","Data":"bc88fda8add3534eb4b197eec8c684f4f0f9feb47ecf4d645a2beae27d0ca46f"} Nov 03 22:52:38.928844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.928798 4903 scope.go:122] "RemoveContainer" containerID="2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2" Nov 03 22:52:38.930500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.930467 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" event={"ID":"2d6bf448-b4ff-4082-a31f-c7a9062769e8","Type":"ContainerStarted","Data":"d2594a51aab6a6b5797e0f63e549d07f1df6deb042fdc7bf2ec5d2dad6929262"} Nov 03 22:52:38.940417 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.940402 4903 scope.go:122] "RemoveContainer" containerID="2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2" Nov 03 22:52:38.940702 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:52:38.940683 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2\": container with ID starting with 2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2 not found: ID does not exist" containerID="2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2" Nov 03 22:52:38.940781 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.940714 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2"} err="failed to get container status \"2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2\": rpc error: code = NotFound desc = could not find container \"2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2\": container with ID starting with 2b935b51c682586444148191b3bf99517d1575da36605fd4d6b6062678faebe2 not found: ID does not exist" Nov 03 22:52:38.978883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.978849 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs"] Nov 03 22:52:38.982906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:38.982882 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fe7ca0-55488976b6-5hnxs"] Nov 03 22:52:39.735767 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:39.735692 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb7b18e-b8d1-45dd-99db-999de387124c" path="/var/lib/kubelet/pods/bbb7b18e-b8d1-45dd-99db-999de387124c/volumes" Nov 03 22:52:39.940104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:39.940068 4903 generic.go:356] "Generic (PLEG): container finished" podID="2d6bf448-b4ff-4082-a31f-c7a9062769e8" containerID="7ba2d07ea61bfe63a83237b086d9aa50fa167c16f1ff7dacff00d4624f848522" exitCode=0 Nov 03 22:52:39.940334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:39.940150 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" event={"ID":"2d6bf448-b4ff-4082-a31f-c7a9062769e8","Type":"ContainerDied","Data":"7ba2d07ea61bfe63a83237b086d9aa50fa167c16f1ff7dacff00d4624f848522"} Nov 03 22:52:41.128000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.127972 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:41.239724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.239639 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6bf448-b4ff-4082-a31f-c7a9062769e8-kube-api-access-pwv26\" (UniqueName: \"kubernetes.io/projected/2d6bf448-b4ff-4082-a31f-c7a9062769e8-kube-api-access-pwv26\") pod \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " Nov 03 22:52:41.239890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.239728 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-xks-ca\") pod \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " Nov 03 22:52:41.239890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.239835 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-openshift-ca\") pod \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\" (UID: \"2d6bf448-b4ff-4082-a31f-c7a9062769e8\") " Nov 03 22:52:41.240175 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.240150 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-openshift-ca" pod "2d6bf448-b4ff-4082-a31f-c7a9062769e8" (UID: "2d6bf448-b4ff-4082-a31f-c7a9062769e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:52:41.240474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.240453 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:41.241759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.241738 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-xks-ca" pod "2d6bf448-b4ff-4082-a31f-c7a9062769e8" (UID: "2d6bf448-b4ff-4082-a31f-c7a9062769e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:41.241841 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.241822 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6bf448-b4ff-4082-a31f-c7a9062769e8-kube-api-access-pwv26" pod "2d6bf448-b4ff-4082-a31f-c7a9062769e8" (UID: "2d6bf448-b4ff-4082-a31f-c7a9062769e8"). InnerVolumeSpecName "kube-api-access-pwv26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:41.342286 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.342253 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwv26\" (UniqueName: \"kubernetes.io/projected/2d6bf448-b4ff-4082-a31f-c7a9062769e8-kube-api-access-pwv26\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:41.342286 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.342280 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6bf448-b4ff-4082-a31f-c7a9062769e8-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:41.958712 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.958679 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" Nov 03 22:52:41.958917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.958712 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p" event={"ID":"2d6bf448-b4ff-4082-a31f-c7a9062769e8","Type":"ContainerDied","Data":"d2594a51aab6a6b5797e0f63e549d07f1df6deb042fdc7bf2ec5d2dad6929262"} Nov 03 22:52:41.958917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:41.958760 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2594a51aab6a6b5797e0f63e549d07f1df6deb042fdc7bf2ec5d2dad6929262" Nov 03 22:52:48.386551 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:52:48.386505 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:52:52.451337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.450962 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" podUID="09059d58-da66-4317-a7cf-a230f29333bc" containerName="main" containerID="cri-o://546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7" gracePeriod=45 Nov 03 22:52:52.770218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.770190 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:52:52.827342 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.827307 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-home\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-home\") pod \"09059d58-da66-4317-a7cf-a230f29333bc\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " Nov 03 22:52:52.827522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.827383 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09059d58-da66-4317-a7cf-a230f29333bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/09059d58-da66-4317-a7cf-a230f29333bc-tls-certs\") pod \"09059d58-da66-4317-a7cf-a230f29333bc\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " Nov 03 22:52:52.827522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.827408 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09059d58-da66-4317-a7cf-a230f29333bc-kube-api-access-fqfxw\" (UniqueName: \"kubernetes.io/projected/09059d58-da66-4317-a7cf-a230f29333bc-kube-api-access-fqfxw\") pod \"09059d58-da66-4317-a7cf-a230f29333bc\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " Nov 03 22:52:52.827648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.827511 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-home" pod "09059d58-da66-4317-a7cf-a230f29333bc" (UID: "09059d58-da66-4317-a7cf-a230f29333bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:52.827648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.827561 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-model-cache\") pod \"09059d58-da66-4317-a7cf-a230f29333bc\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " Nov 03 22:52:52.827648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.827603 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-tmp-dir\") pod \"09059d58-da66-4317-a7cf-a230f29333bc\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " Nov 03 22:52:52.827806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.827646 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-dshm\") pod \"09059d58-da66-4317-a7cf-a230f29333bc\" (UID: \"09059d58-da66-4317-a7cf-a230f29333bc\") " Nov 03 22:52:52.827861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.827800 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-model-cache" pod "09059d58-da66-4317-a7cf-a230f29333bc" (UID: "09059d58-da66-4317-a7cf-a230f29333bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:52.827939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.827913 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-tmp-dir" pod "09059d58-da66-4317-a7cf-a230f29333bc" (UID: "09059d58-da66-4317-a7cf-a230f29333bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:52.828911 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.828886 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:52.829052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.828913 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:52.829052 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.828927 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:52.830408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.830385 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09059d58-da66-4317-a7cf-a230f29333bc-tls-certs" pod "09059d58-da66-4317-a7cf-a230f29333bc" (UID: "09059d58-da66-4317-a7cf-a230f29333bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:52:52.830659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.830637 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09059d58-da66-4317-a7cf-a230f29333bc-kube-api-access-fqfxw" pod "09059d58-da66-4317-a7cf-a230f29333bc" (UID: "09059d58-da66-4317-a7cf-a230f29333bc"). InnerVolumeSpecName "kube-api-access-fqfxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:52:52.830754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.830705 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-dshm" pod "09059d58-da66-4317-a7cf-a230f29333bc" (UID: "09059d58-da66-4317-a7cf-a230f29333bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:52:52.930583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.930548 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09059d58-da66-4317-a7cf-a230f29333bc-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:52.930583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.930582 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfxw\" (UniqueName: \"kubernetes.io/projected/09059d58-da66-4317-a7cf-a230f29333bc-kube-api-access-fqfxw\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:52.930790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:52.930594 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09059d58-da66-4317-a7cf-a230f29333bc-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:52:53.051842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.051761 4903 generic.go:356] "Generic (PLEG): container finished" podID="09059d58-da66-4317-a7cf-a230f29333bc" containerID="546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7" exitCode=0 Nov 03 22:52:53.051842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.051817 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" Nov 03 22:52:53.051842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.051831 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" event={"ID":"09059d58-da66-4317-a7cf-a230f29333bc","Type":"ContainerDied","Data":"546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7"} Nov 03 22:52:53.052132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.051860 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b" event={"ID":"09059d58-da66-4317-a7cf-a230f29333bc","Type":"ContainerDied","Data":"ec85d264871fa910253bddd701c64154a45bc665a9c1f267c89f1af9cd4cebb0"} Nov 03 22:52:53.052132 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.051890 4903 scope.go:122] "RemoveContainer" containerID="546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7" Nov 03 22:52:53.067109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.067085 4903 scope.go:122] "RemoveContainer" containerID="546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7" Nov 03 22:52:53.067435 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:52:53.067409 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7\": container with ID starting with 546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7 not found: ID does not exist" containerID="546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7" Nov 03 22:52:53.067554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.067442 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7"} err="failed to get container status \"546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7\": rpc error: code = NotFound desc = could not find container \"546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7\": container with ID starting with 546298c2aebeee217607bda3eb047babb358652ce9d3e176c6a21a359fbb3ee7 not found: ID does not exist" Nov 03 22:52:53.102410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.102379 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b"] Nov 03 22:52:53.108970 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.108944 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-341c51/test-model-341c51-kserve-6977dcb48f-rrv9b"] Nov 03 22:52:53.738590 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:53.738550 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09059d58-da66-4317-a7cf-a230f29333bc" path="/var/lib/kubelet/pods/09059d58-da66-4317-a7cf-a230f29333bc/volumes" Nov 03 22:52:54.266423 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:54.266394 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:52:54.268236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:54.268211 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:52:54.275506 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:54.275483 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:52:54.277281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:54.277263 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:52:55.039954 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:55.039917 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59"] Nov 03 22:52:55.049676 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:55.049648 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4865466c-mqs59"] Nov 03 22:52:55.738547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:55.738515 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a68a75c6-1469-4ce8-b300-e010e6b48b19" path="/var/lib/kubelet/pods/a68a75c6-1469-4ce8-b300-e010e6b48b19/volumes" Nov 03 22:52:58.795845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:58.795806 4903 scope.go:122] "RemoveContainer" containerID="dd322b70a566c8cb17cd27bf827160d8e57942fc69adeb9fd1cac5a2ad48a83f" Nov 03 22:52:58.807578 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:52:58.807557 4903 scope.go:122] "RemoveContainer" containerID="cc203e7843861c0b7f42622c2ff6b6b54d184ad59f8e27b208c65b47578b96e7" Nov 03 22:53:04.825697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:04.825657 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp"] Nov 03 22:53:04.834378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:04.834346 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fe7ca0-br5qp"] Nov 03 22:53:05.737881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:05.737845 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a0d143-82bc-4866-95fe-1f757a010762" path="/var/lib/kubelet/pods/f8a0d143-82bc-4866-95fe-1f757a010762/volumes" Nov 03 22:53:15.898752 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:15.898713 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4"] Nov 03 22:53:16.232059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.231993 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t"] Nov 03 22:53:16.233919 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.233889 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2d6bf448-b4ff-4082-a31f-c7a9062769e8" containerName="revoke-keys" Nov 03 22:53:16.234085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.234069 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6bf448-b4ff-4082-a31f-c7a9062769e8" containerName="revoke-keys" Nov 03 22:53:16.234218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.234206 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bbb7b18e-b8d1-45dd-99db-999de387124c" containerName="payload-processing" Nov 03 22:53:16.234396 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.234381 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb7b18e-b8d1-45dd-99db-999de387124c" containerName="payload-processing" Nov 03 22:53:16.234525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.234514 4903 cpu_manager.go:414] "containerMap: removing container" podUID="09059d58-da66-4317-a7cf-a230f29333bc" containerName="main" Nov 03 22:53:16.234600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.234590 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09059d58-da66-4317-a7cf-a230f29333bc" containerName="main" Nov 03 22:53:16.235087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.235067 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6bf448-b4ff-4082-a31f-c7a9062769e8" containerName="revoke-keys" Nov 03 22:53:16.235253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.235228 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb7b18e-b8d1-45dd-99db-999de387124c" containerName="payload-processing" Nov 03 22:53:16.235471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.235261 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="09059d58-da66-4317-a7cf-a230f29333bc" containerName="main" Nov 03 22:53:16.242022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.241997 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.251613 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.251587 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t"] Nov 03 22:53:16.255871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.255851 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:53:16.256540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.256521 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:53:16.278561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.278526 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:53:16.278561 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.278552 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:53:16.400653 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.400615 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-341c51-n948t\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.400857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.400833 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-341c51-n948t\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.400916 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.400873 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdd9r\" (UniqueName: \"kubernetes.io/projected/63d13c68-2f0a-4a75-803d-8735440f7f91-kube-api-access-qdd9r\") pod \"maas-api-revoke-keys-e2e-inf-a-341c51-n948t\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.503738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.502479 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-341c51-n948t\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.503738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.502543 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdd9r\" (UniqueName: \"kubernetes.io/projected/63d13c68-2f0a-4a75-803d-8735440f7f91-kube-api-access-qdd9r\") pod \"maas-api-revoke-keys-e2e-inf-a-341c51-n948t\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.503738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.502688 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-341c51-n948t\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.504128 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.503924 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-341c51-n948t\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.513008 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.512968 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-341c51-n948t\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.526025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.525994 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdd9r\" (UniqueName: \"kubernetes.io/projected/63d13c68-2f0a-4a75-803d-8735440f7f91-kube-api-access-qdd9r\") pod \"maas-api-revoke-keys-e2e-inf-a-341c51-n948t\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.562812 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.562787 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:16.872333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:16.872281 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t"] Nov 03 22:53:16.873487 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:53:16.873460 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d13c68_2f0a_4a75_803d_8735440f7f91.slice/crio-3ac7c676b13e4c521aaf1591a91a01b4d33da912a0445c011aa401d526832a80 WatchSource:0}: Error finding container 3ac7c676b13e4c521aaf1591a91a01b4d33da912a0445c011aa401d526832a80: Status 404 returned error can't find the container with id 3ac7c676b13e4c521aaf1591a91a01b4d33da912a0445c011aa401d526832a80 Nov 03 22:53:17.034978 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:17.034940 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9"] Nov 03 22:53:17.038786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:17.038755 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fca835c-ggmt9"] Nov 03 22:53:17.313609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:17.302629 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" event={"ID":"63d13c68-2f0a-4a75-803d-8735440f7f91","Type":"ContainerStarted","Data":"1ba9b57d91c1044a609e35ebcb92306bd338b1f0f530e7d965f2446077478fc3"} Nov 03 22:53:17.313609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:17.302696 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" event={"ID":"63d13c68-2f0a-4a75-803d-8735440f7f91","Type":"ContainerStarted","Data":"3ac7c676b13e4c521aaf1591a91a01b4d33da912a0445c011aa401d526832a80"} Nov 03 22:53:17.341123 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:17.341057 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" podStartSLOduration=1.341038374 podStartE2EDuration="1.341038374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:17.340246491 +0000 UTC m=+2423.744187467" watchObservedRunningTime="2026-11-03 22:53:17.341038374 +0000 UTC m=+2423.744979348" Nov 03 22:53:17.738079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:17.737993 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125cfca1-f3d2-4ca7-8c21-f6509297e54f" path="/var/lib/kubelet/pods/125cfca1-f3d2-4ca7-8c21-f6509297e54f/volumes" Nov 03 22:53:18.312922 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:18.312886 4903 generic.go:356] "Generic (PLEG): container finished" podID="63d13c68-2f0a-4a75-803d-8735440f7f91" containerID="1ba9b57d91c1044a609e35ebcb92306bd338b1f0f530e7d965f2446077478fc3" exitCode=0 Nov 03 22:53:18.313363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:18.312965 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" event={"ID":"63d13c68-2f0a-4a75-803d-8735440f7f91","Type":"ContainerDied","Data":"1ba9b57d91c1044a609e35ebcb92306bd338b1f0f530e7d965f2446077478fc3"} Nov 03 22:53:18.387789 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:53:18.387749 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:53:19.158921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.158890 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r"] Nov 03 22:53:19.163593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.163576 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.181373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.181348 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r"] Nov 03 22:53:19.241704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.241664 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f9f8b1-fld7r\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.241896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.241717 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289hh\" (UniqueName: \"kubernetes.io/projected/bc7407c2-e51e-4bfb-aaa7-160ce693340d-kube-api-access-289hh\") pod \"maas-api-revoke-sub-f1f9f8b1-fld7r\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.241896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.241815 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f9f8b1-fld7r\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.343635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.343581 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f9f8b1-fld7r\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.344074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.343677 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289hh\" (UniqueName: \"kubernetes.io/projected/bc7407c2-e51e-4bfb-aaa7-160ce693340d-kube-api-access-289hh\") pod \"maas-api-revoke-sub-f1f9f8b1-fld7r\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.344074 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.343774 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f9f8b1-fld7r\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.344252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.344231 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f9f8b1-fld7r\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.346113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.346089 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f9f8b1-fld7r\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.370212 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.370179 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-289hh\" (UniqueName: \"kubernetes.io/projected/bc7407c2-e51e-4bfb-aaa7-160ce693340d-kube-api-access-289hh\") pod \"maas-api-revoke-sub-f1f9f8b1-fld7r\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.477601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.477571 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:19.521802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.521770 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:19.647980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.647949 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-xks-ca\") pod \"63d13c68-2f0a-4a75-803d-8735440f7f91\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " Nov 03 22:53:19.648180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.648157 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-openshift-ca\") pod \"63d13c68-2f0a-4a75-803d-8735440f7f91\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " Nov 03 22:53:19.648248 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.648228 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d13c68-2f0a-4a75-803d-8735440f7f91-kube-api-access-qdd9r\" (UniqueName: \"kubernetes.io/projected/63d13c68-2f0a-4a75-803d-8735440f7f91-kube-api-access-qdd9r\") pod \"63d13c68-2f0a-4a75-803d-8735440f7f91\" (UID: \"63d13c68-2f0a-4a75-803d-8735440f7f91\") " Nov 03 22:53:19.648593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.648567 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-openshift-ca" pod "63d13c68-2f0a-4a75-803d-8735440f7f91" (UID: "63d13c68-2f0a-4a75-803d-8735440f7f91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:19.648893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.648872 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:19.650093 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.650069 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-xks-ca" pod "63d13c68-2f0a-4a75-803d-8735440f7f91" (UID: "63d13c68-2f0a-4a75-803d-8735440f7f91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:19.650447 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.650425 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d13c68-2f0a-4a75-803d-8735440f7f91-kube-api-access-qdd9r" pod "63d13c68-2f0a-4a75-803d-8735440f7f91" (UID: "63d13c68-2f0a-4a75-803d-8735440f7f91"). InnerVolumeSpecName "kube-api-access-qdd9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:19.750469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.750077 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdd9r\" (UniqueName: \"kubernetes.io/projected/63d13c68-2f0a-4a75-803d-8735440f7f91-kube-api-access-qdd9r\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:19.750771 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.750736 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63d13c68-2f0a-4a75-803d-8735440f7f91-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:19.796413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:19.796383 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r"] Nov 03 22:53:19.796542 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:53:19.796427 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7407c2_e51e_4bfb_aaa7_160ce693340d.slice/crio-78e25c7193c6b7be0f0199fda1d49d274be3c6491f1df2c2605ace7296de93c1 WatchSource:0}: Error finding container 78e25c7193c6b7be0f0199fda1d49d274be3c6491f1df2c2605ace7296de93c1: Status 404 returned error can't find the container with id 78e25c7193c6b7be0f0199fda1d49d274be3c6491f1df2c2605ace7296de93c1 Nov 03 22:53:20.331439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:20.331342 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" event={"ID":"bc7407c2-e51e-4bfb-aaa7-160ce693340d","Type":"ContainerStarted","Data":"29bbd7562a585778e30da2b62147d203f2a99f75d508b5f5151dd12a8a00d1f1"} Nov 03 22:53:20.331606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:20.331446 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" event={"ID":"bc7407c2-e51e-4bfb-aaa7-160ce693340d","Type":"ContainerStarted","Data":"78e25c7193c6b7be0f0199fda1d49d274be3c6491f1df2c2605ace7296de93c1"} Nov 03 22:53:20.332867 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:20.332844 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" Nov 03 22:53:20.332996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:20.332843 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t" event={"ID":"63d13c68-2f0a-4a75-803d-8735440f7f91","Type":"ContainerDied","Data":"3ac7c676b13e4c521aaf1591a91a01b4d33da912a0445c011aa401d526832a80"} Nov 03 22:53:20.332996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:20.332966 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ac7c676b13e4c521aaf1591a91a01b4d33da912a0445c011aa401d526832a80" Nov 03 22:53:20.371366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:20.371319 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" podStartSLOduration=1.371289169 podStartE2EDuration="1.371289169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:20.369765413 +0000 UTC m=+2426.773706389" watchObservedRunningTime="2026-11-03 22:53:20.371289169 +0000 UTC m=+2426.775230161" Nov 03 22:53:21.342171 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:21.342138 4903 generic.go:356] "Generic (PLEG): container finished" podID="bc7407c2-e51e-4bfb-aaa7-160ce693340d" containerID="29bbd7562a585778e30da2b62147d203f2a99f75d508b5f5151dd12a8a00d1f1" exitCode=0 Nov 03 22:53:21.342397 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:21.342219 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" event={"ID":"bc7407c2-e51e-4bfb-aaa7-160ce693340d","Type":"ContainerDied","Data":"29bbd7562a585778e30da2b62147d203f2a99f75d508b5f5151dd12a8a00d1f1"} Nov 03 22:53:22.528456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.528431 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:22.692286 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.692254 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7407c2-e51e-4bfb-aaa7-160ce693340d-kube-api-access-289hh\" (UniqueName: \"kubernetes.io/projected/bc7407c2-e51e-4bfb-aaa7-160ce693340d-kube-api-access-289hh\") pod \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " Nov 03 22:53:22.692482 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.692380 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-xks-ca\") pod \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " Nov 03 22:53:22.692547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.692536 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-openshift-ca\") pod \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\" (UID: \"bc7407c2-e51e-4bfb-aaa7-160ce693340d\") " Nov 03 22:53:22.692897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.692868 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-openshift-ca" pod "bc7407c2-e51e-4bfb-aaa7-160ce693340d" (UID: "bc7407c2-e51e-4bfb-aaa7-160ce693340d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:53:22.693046 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.692998 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:22.694522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.694492 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-xks-ca" pod "bc7407c2-e51e-4bfb-aaa7-160ce693340d" (UID: "bc7407c2-e51e-4bfb-aaa7-160ce693340d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:22.694626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.694582 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7407c2-e51e-4bfb-aaa7-160ce693340d-kube-api-access-289hh" pod "bc7407c2-e51e-4bfb-aaa7-160ce693340d" (UID: "bc7407c2-e51e-4bfb-aaa7-160ce693340d"). InnerVolumeSpecName "kube-api-access-289hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:22.794288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.794251 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-289hh\" (UniqueName: \"kubernetes.io/projected/bc7407c2-e51e-4bfb-aaa7-160ce693340d-kube-api-access-289hh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:22.794288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:22.794282 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7407c2-e51e-4bfb-aaa7-160ce693340d-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:23.358514 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:23.358484 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" Nov 03 22:53:23.358697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:23.358517 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r" event={"ID":"bc7407c2-e51e-4bfb-aaa7-160ce693340d","Type":"ContainerDied","Data":"78e25c7193c6b7be0f0199fda1d49d274be3c6491f1df2c2605ace7296de93c1"} Nov 03 22:53:23.358697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:23.358565 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78e25c7193c6b7be0f0199fda1d49d274be3c6491f1df2c2605ace7296de93c1" Nov 03 22:53:23.625232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:23.625144 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:53:30.925118 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:30.925079 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" podUID="e91da8b1-f0a1-439c-9643-078c90f8ea5e" containerName="main" containerID="cri-o://141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66" gracePeriod=45 Nov 03 22:53:31.039738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.039703 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" podUID="e91da8b1-f0a1-439c-9643-078c90f8ea5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.255:8000/ready\": dial tcp 10.132.0.255:8000: connect: connection refused" Nov 03 22:53:31.243951 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.243926 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:53:31.314688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.314654 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-dshm\") pod \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " Nov 03 22:53:31.314899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.314703 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tmp-dir\") pod \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " Nov 03 22:53:31.314899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.314757 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91da8b1-f0a1-439c-9643-078c90f8ea5e-kube-api-access-4wfm9\" (UniqueName: \"kubernetes.io/projected/e91da8b1-f0a1-439c-9643-078c90f8ea5e-kube-api-access-4wfm9\") pod \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " Nov 03 22:53:31.314899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.314778 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-home\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-home\") pod \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " Nov 03 22:53:31.315084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.314957 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tls-certs\") pod \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " Nov 03 22:53:31.315084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.315054 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-model-cache\") pod \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\" (UID: \"e91da8b1-f0a1-439c-9643-078c90f8ea5e\") " Nov 03 22:53:31.315084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.315053 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tmp-dir" pod "e91da8b1-f0a1-439c-9643-078c90f8ea5e" (UID: "e91da8b1-f0a1-439c-9643-078c90f8ea5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:31.315819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.315225 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-home" pod "e91da8b1-f0a1-439c-9643-078c90f8ea5e" (UID: "e91da8b1-f0a1-439c-9643-078c90f8ea5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:31.315819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.315433 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-model-cache" pod "e91da8b1-f0a1-439c-9643-078c90f8ea5e" (UID: "e91da8b1-f0a1-439c-9643-078c90f8ea5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:31.315819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.315643 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:31.315819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.315665 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:31.315819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.315678 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:31.317595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.317557 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91da8b1-f0a1-439c-9643-078c90f8ea5e-kube-api-access-4wfm9" pod "e91da8b1-f0a1-439c-9643-078c90f8ea5e" (UID: "e91da8b1-f0a1-439c-9643-078c90f8ea5e"). InnerVolumeSpecName "kube-api-access-4wfm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:53:31.317727 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.317701 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tls-certs" pod "e91da8b1-f0a1-439c-9643-078c90f8ea5e" (UID: "e91da8b1-f0a1-439c-9643-078c90f8ea5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:53:31.317806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.317727 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-dshm" pod "e91da8b1-f0a1-439c-9643-078c90f8ea5e" (UID: "e91da8b1-f0a1-439c-9643-078c90f8ea5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:53:31.417466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.417432 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e91da8b1-f0a1-439c-9643-078c90f8ea5e-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:31.417466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.417462 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wfm9\" (UniqueName: \"kubernetes.io/projected/e91da8b1-f0a1-439c-9643-078c90f8ea5e-kube-api-access-4wfm9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:31.417466 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.417473 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e91da8b1-f0a1-439c-9643-078c90f8ea5e-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:53:31.428828 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.428798 4903 generic.go:356] "Generic (PLEG): container finished" podID="e91da8b1-f0a1-439c-9643-078c90f8ea5e" containerID="141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66" exitCode=0 Nov 03 22:53:31.428993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.428846 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" Nov 03 22:53:31.428993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.428882 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" event={"ID":"e91da8b1-f0a1-439c-9643-078c90f8ea5e","Type":"ContainerDied","Data":"141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66"} Nov 03 22:53:31.428993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.428927 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4" event={"ID":"e91da8b1-f0a1-439c-9643-078c90f8ea5e","Type":"ContainerDied","Data":"004e90cd8175afa8def0d3531caefa9bbe8af4e3a1c19103feca7b84b1a988a8"} Nov 03 22:53:31.428993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.428944 4903 scope.go:122] "RemoveContainer" containerID="141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66" Nov 03 22:53:31.441270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.441248 4903 scope.go:122] "RemoveContainer" containerID="141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66" Nov 03 22:53:31.441586 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:53:31.441565 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66\": container with ID starting with 141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66 not found: ID does not exist" containerID="141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66" Nov 03 22:53:31.441649 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.441594 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66"} err="failed to get container status \"141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66\": rpc error: code = NotFound desc = could not find container \"141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66\": container with ID starting with 141a809edefc2e30f8b3e4c403a81b163efa9af81b8a7575dc8ec7d673414b66 not found: ID does not exist" Nov 03 22:53:31.549723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.549644 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4"] Nov 03 22:53:31.565902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.565865 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-124490/test-model-124490-kserve-6745d68f47-kxwp4"] Nov 03 22:53:31.752733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:31.752697 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e91da8b1-f0a1-439c-9643-078c90f8ea5e" path="/var/lib/kubelet/pods/e91da8b1-f0a1-439c-9643-078c90f8ea5e/volumes" Nov 03 22:53:48.385246 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:53:48.385204 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:53:56.973026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:56.972990 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw"] Nov 03 22:53:57.556917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.556881 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx"] Nov 03 22:53:57.557652 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.557637 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bc7407c2-e51e-4bfb-aaa7-160ce693340d" containerName="revoke-keys" Nov 03 22:53:57.557730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.557652 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7407c2-e51e-4bfb-aaa7-160ce693340d" containerName="revoke-keys" Nov 03 22:53:57.557730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.557667 4903 cpu_manager.go:414] "containerMap: removing container" podUID="63d13c68-2f0a-4a75-803d-8735440f7f91" containerName="revoke-keys" Nov 03 22:53:57.557730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.557672 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d13c68-2f0a-4a75-803d-8735440f7f91" containerName="revoke-keys" Nov 03 22:53:57.557730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.557689 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e91da8b1-f0a1-439c-9643-078c90f8ea5e" containerName="main" Nov 03 22:53:57.557730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.557694 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e91da8b1-f0a1-439c-9643-078c90f8ea5e" containerName="main" Nov 03 22:53:57.557896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.557799 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7407c2-e51e-4bfb-aaa7-160ce693340d" containerName="revoke-keys" Nov 03 22:53:57.557896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.557809 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d13c68-2f0a-4a75-803d-8735440f7f91" containerName="revoke-keys" Nov 03 22:53:57.557896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.557817 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e91da8b1-f0a1-439c-9643-078c90f8ea5e" containerName="main" Nov 03 22:53:57.561530 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.561512 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.580104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.580069 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx"] Nov 03 22:53:57.584751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.584722 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:53:57.584751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.584734 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:53:57.584945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.584763 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:53:57.584945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.584802 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:53:57.655282 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.655244 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d589901a-d2vhx\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.655490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.655286 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p622\" (UniqueName: \"kubernetes.io/projected/b778c9da-6e2b-4780-b4bc-895754557c01-kube-api-access-9p622\") pod \"maas-api-revoke-sub-d589901a-d2vhx\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.655490 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.655462 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d589901a-d2vhx\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.757047 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.757017 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d589901a-d2vhx\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.757219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.757053 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p622\" (UniqueName: \"kubernetes.io/projected/b778c9da-6e2b-4780-b4bc-895754557c01-kube-api-access-9p622\") pod \"maas-api-revoke-sub-d589901a-d2vhx\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.757219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.757108 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d589901a-d2vhx\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.757789 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.757766 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d589901a-d2vhx\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.759363 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.759345 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d589901a-d2vhx\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.779242 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.779221 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p622\" (UniqueName: \"kubernetes.io/projected/b778c9da-6e2b-4780-b4bc-895754557c01-kube-api-access-9p622\") pod \"maas-api-revoke-sub-d589901a-d2vhx\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:57.875985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:57.875899 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:53:58.186186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:58.186154 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx"] Nov 03 22:53:58.187444 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:53:58.187417 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb778c9da_6e2b_4780_b4bc_895754557c01.slice/crio-be66f2c0efa8a9359696b505c1b7381599be23c2eb401d608bc986deb236f5a9 WatchSource:0}: Error finding container be66f2c0efa8a9359696b505c1b7381599be23c2eb401d608bc986deb236f5a9: Status 404 returned error can't find the container with id be66f2c0efa8a9359696b505c1b7381599be23c2eb401d608bc986deb236f5a9 Nov 03 22:53:58.666160 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:58.666129 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" event={"ID":"b778c9da-6e2b-4780-b4bc-895754557c01","Type":"ContainerStarted","Data":"090e966ee10c469a466bfc2a7f0ea02c85430cdd2e13a5dc501af02f7a174c8d"} Nov 03 22:53:58.666309 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:58.666183 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" event={"ID":"b778c9da-6e2b-4780-b4bc-895754557c01","Type":"ContainerStarted","Data":"be66f2c0efa8a9359696b505c1b7381599be23c2eb401d608bc986deb236f5a9"} Nov 03 22:53:58.706968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:58.706918 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" podStartSLOduration=1.7069030170000001 podStartE2EDuration="1.706903017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:53:58.704660031 +0000 UTC m=+2465.108601009" watchObservedRunningTime="2026-11-03 22:53:58.706903017 +0000 UTC m=+2465.110844031" Nov 03 22:53:58.876774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:58.876693 4903 scope.go:122] "RemoveContainer" containerID="c5328bac83b80ef56198cf5c7d4955df1eee6047efa0ef39220d6d158855cbd8" Nov 03 22:53:58.887626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:58.887396 4903 scope.go:122] "RemoveContainer" containerID="812fb1afdd7d9105107594a85b272d036322ac6c3e3fbd35b110ee16e8396815" Nov 03 22:53:58.896820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:58.896802 4903 scope.go:122] "RemoveContainer" containerID="4f8fe78fb152079d45d8a8a82b2ba88fc6bbeee0bba592d9e9e736b35629097b" Nov 03 22:53:59.676900 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:59.676867 4903 generic.go:356] "Generic (PLEG): container finished" podID="b778c9da-6e2b-4780-b4bc-895754557c01" containerID="090e966ee10c469a466bfc2a7f0ea02c85430cdd2e13a5dc501af02f7a174c8d" exitCode=0 Nov 03 22:53:59.677333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:53:59.676943 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" event={"ID":"b778c9da-6e2b-4780-b4bc-895754557c01","Type":"ContainerDied","Data":"090e966ee10c469a466bfc2a7f0ea02c85430cdd2e13a5dc501af02f7a174c8d"} Nov 03 22:54:00.871824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:00.871798 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:54:00.909481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:00.909453 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-openshift-ca\") pod \"b778c9da-6e2b-4780-b4bc-895754557c01\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " Nov 03 22:54:00.909650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:00.909503 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-xks-ca\") pod \"b778c9da-6e2b-4780-b4bc-895754557c01\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " Nov 03 22:54:00.909650 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:00.909526 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b778c9da-6e2b-4780-b4bc-895754557c01-kube-api-access-9p622\" (UniqueName: \"kubernetes.io/projected/b778c9da-6e2b-4780-b4bc-895754557c01-kube-api-access-9p622\") pod \"b778c9da-6e2b-4780-b4bc-895754557c01\" (UID: \"b778c9da-6e2b-4780-b4bc-895754557c01\") " Nov 03 22:54:00.909880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:00.909857 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-openshift-ca" pod "b778c9da-6e2b-4780-b4bc-895754557c01" (UID: "b778c9da-6e2b-4780-b4bc-895754557c01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:00.910195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:00.910177 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:00.912089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:00.912031 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-xks-ca" pod "b778c9da-6e2b-4780-b4bc-895754557c01" (UID: "b778c9da-6e2b-4780-b4bc-895754557c01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:00.912255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:00.912065 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b778c9da-6e2b-4780-b4bc-895754557c01-kube-api-access-9p622" pod "b778c9da-6e2b-4780-b4bc-895754557c01" (UID: "b778c9da-6e2b-4780-b4bc-895754557c01"). InnerVolumeSpecName "kube-api-access-9p622". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:01.011630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:01.011542 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b778c9da-6e2b-4780-b4bc-895754557c01-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:01.011630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:01.011577 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p622\" (UniqueName: \"kubernetes.io/projected/b778c9da-6e2b-4780-b4bc-895754557c01-kube-api-access-9p622\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:01.694307 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:01.694273 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" event={"ID":"b778c9da-6e2b-4780-b4bc-895754557c01","Type":"ContainerDied","Data":"be66f2c0efa8a9359696b505c1b7381599be23c2eb401d608bc986deb236f5a9"} Nov 03 22:54:01.694498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:01.694315 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx" Nov 03 22:54:01.694498 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:01.694345 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be66f2c0efa8a9359696b505c1b7381599be23c2eb401d608bc986deb236f5a9" Nov 03 22:54:05.012312 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:05.012261 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t"] Nov 03 22:54:05.021967 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:05.021940 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-341c51-n948t"] Nov 03 22:54:05.738178 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:05.738143 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d13c68-2f0a-4a75-803d-8735440f7f91" path="/var/lib/kubelet/pods/63d13c68-2f0a-4a75-803d-8735440f7f91/volumes" Nov 03 22:54:11.996976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:11.996935 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" podUID="a1527d9f-2f08-4e37-88ea-22bbca92705b" containerName="main" containerID="cri-o://1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba" gracePeriod=45 Nov 03 22:54:12.273758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.273734 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:54:12.407161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.407129 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-home\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-home\") pod \"a1527d9f-2f08-4e37-88ea-22bbca92705b\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " Nov 03 22:54:12.407368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.407266 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1527d9f-2f08-4e37-88ea-22bbca92705b-kube-api-access-zl5p4\" (UniqueName: \"kubernetes.io/projected/a1527d9f-2f08-4e37-88ea-22bbca92705b-kube-api-access-zl5p4\") pod \"a1527d9f-2f08-4e37-88ea-22bbca92705b\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " Nov 03 22:54:12.407368 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.407325 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-tmp-dir\") pod \"a1527d9f-2f08-4e37-88ea-22bbca92705b\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " Nov 03 22:54:12.407528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.407415 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1527d9f-2f08-4e37-88ea-22bbca92705b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1527d9f-2f08-4e37-88ea-22bbca92705b-tls-certs\") pod \"a1527d9f-2f08-4e37-88ea-22bbca92705b\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " Nov 03 22:54:12.407528 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.407487 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-dshm\") pod \"a1527d9f-2f08-4e37-88ea-22bbca92705b\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " Nov 03 22:54:12.407648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.407496 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-home" pod "a1527d9f-2f08-4e37-88ea-22bbca92705b" (UID: "a1527d9f-2f08-4e37-88ea-22bbca92705b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:12.407648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.407571 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-model-cache\") pod \"a1527d9f-2f08-4e37-88ea-22bbca92705b\" (UID: \"a1527d9f-2f08-4e37-88ea-22bbca92705b\") " Nov 03 22:54:12.407648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.407582 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-tmp-dir" pod "a1527d9f-2f08-4e37-88ea-22bbca92705b" (UID: "a1527d9f-2f08-4e37-88ea-22bbca92705b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:12.408113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.408069 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-model-cache" pod "a1527d9f-2f08-4e37-88ea-22bbca92705b" (UID: "a1527d9f-2f08-4e37-88ea-22bbca92705b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:12.408357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.408130 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:12.408467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.408366 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:12.409787 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.409737 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1527d9f-2f08-4e37-88ea-22bbca92705b-tls-certs" pod "a1527d9f-2f08-4e37-88ea-22bbca92705b" (UID: "a1527d9f-2f08-4e37-88ea-22bbca92705b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:12.409959 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.409934 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-dshm" pod "a1527d9f-2f08-4e37-88ea-22bbca92705b" (UID: "a1527d9f-2f08-4e37-88ea-22bbca92705b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:12.410011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.409954 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1527d9f-2f08-4e37-88ea-22bbca92705b-kube-api-access-zl5p4" pod "a1527d9f-2f08-4e37-88ea-22bbca92705b" (UID: "a1527d9f-2f08-4e37-88ea-22bbca92705b"). InnerVolumeSpecName "kube-api-access-zl5p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:12.510269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.510183 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:12.510269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.510215 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1527d9f-2f08-4e37-88ea-22bbca92705b-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:12.510269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.510227 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl5p4\" (UniqueName: \"kubernetes.io/projected/a1527d9f-2f08-4e37-88ea-22bbca92705b-kube-api-access-zl5p4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:12.510269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.510236 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1527d9f-2f08-4e37-88ea-22bbca92705b-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:12.805409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.805316 4903 generic.go:356] "Generic (PLEG): container finished" podID="a1527d9f-2f08-4e37-88ea-22bbca92705b" containerID="1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba" exitCode=0 Nov 03 22:54:12.805938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.805907 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" event={"ID":"a1527d9f-2f08-4e37-88ea-22bbca92705b","Type":"ContainerDied","Data":"1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba"} Nov 03 22:54:12.806064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.805959 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" event={"ID":"a1527d9f-2f08-4e37-88ea-22bbca92705b","Type":"ContainerDied","Data":"9541d1475e6f98ceeb0a69f11a1f0957e30906d7792893415a5f374ae3b08d51"} Nov 03 22:54:12.806064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.805984 4903 scope.go:122] "RemoveContainer" containerID="1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba" Nov 03 22:54:12.806177 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.806142 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw" Nov 03 22:54:12.830512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.830482 4903 scope.go:122] "RemoveContainer" containerID="1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba" Nov 03 22:54:12.830988 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:12.830954 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba\": container with ID starting with 1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba not found: ID does not exist" containerID="1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba" Nov 03 22:54:12.831124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.831001 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba"} err="failed to get container status \"1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba\": rpc error: code = NotFound desc = could not find container \"1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba\": container with ID starting with 1993ae17094a46cdcd478e11336815d0d0ed2f7b8094d0c44891a9b01658beba not found: ID does not exist" Nov 03 22:54:12.870522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.870488 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw"] Nov 03 22:54:12.874944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:12.874913 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bb1fa5/test-model-bb1fa5-kserve-7d75bcf989-kxhdw"] Nov 03 22:54:13.123897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.123815 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w"] Nov 03 22:54:13.124559 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.124543 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a1527d9f-2f08-4e37-88ea-22bbca92705b" containerName="main" Nov 03 22:54:13.124635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.124560 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1527d9f-2f08-4e37-88ea-22bbca92705b" containerName="main" Nov 03 22:54:13.124635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.124577 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b778c9da-6e2b-4780-b4bc-895754557c01" containerName="revoke-keys" Nov 03 22:54:13.124635 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.124582 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b778c9da-6e2b-4780-b4bc-895754557c01" containerName="revoke-keys" Nov 03 22:54:13.124965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.124686 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1527d9f-2f08-4e37-88ea-22bbca92705b" containerName="main" Nov 03 22:54:13.124965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.124696 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b778c9da-6e2b-4780-b4bc-895754557c01" containerName="revoke-keys" Nov 03 22:54:13.129702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.129684 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.145974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.145931 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:54:13.156942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.156713 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:54:13.158909 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.158112 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:13.162252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.162203 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:54:13.172086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.172056 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w"] Nov 03 22:54:13.220598 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.220561 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-124490-27m9w\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.220819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.220743 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg2ng\" (UniqueName: \"kubernetes.io/projected/e3990d02-5863-49c0-a9f8-8361942d9820-kube-api-access-xg2ng\") pod \"maas-api-revoke-keys-e2e-inf-b-124490-27m9w\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.220819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.220811 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-124490-27m9w\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.322452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.322413 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-124490-27m9w\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.322634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.322525 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg2ng\" (UniqueName: \"kubernetes.io/projected/e3990d02-5863-49c0-a9f8-8361942d9820-kube-api-access-xg2ng\") pod \"maas-api-revoke-keys-e2e-inf-b-124490-27m9w\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.322634 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.322587 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-124490-27m9w\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.323096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.323074 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-124490-27m9w\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.325018 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.324987 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-124490-27m9w\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.343908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.343884 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg2ng\" (UniqueName: \"kubernetes.io/projected/e3990d02-5863-49c0-a9f8-8361942d9820-kube-api-access-xg2ng\") pod \"maas-api-revoke-keys-e2e-inf-b-124490-27m9w\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.466866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.466837 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:13.741278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.741202 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1527d9f-2f08-4e37-88ea-22bbca92705b" path="/var/lib/kubelet/pods/a1527d9f-2f08-4e37-88ea-22bbca92705b/volumes" Nov 03 22:54:13.773240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.773215 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w"] Nov 03 22:54:13.774654 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:54:13.774616 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3990d02_5863_49c0_a9f8_8361942d9820.slice/crio-2503a7221333d7754f7918e4d6ff49ec500d25607ce28d4b9833e454e896f849 WatchSource:0}: Error finding container 2503a7221333d7754f7918e4d6ff49ec500d25607ce28d4b9833e454e896f849: Status 404 returned error can't find the container with id 2503a7221333d7754f7918e4d6ff49ec500d25607ce28d4b9833e454e896f849 Nov 03 22:54:13.815555 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:13.815520 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" event={"ID":"e3990d02-5863-49c0-a9f8-8361942d9820","Type":"ContainerStarted","Data":"2503a7221333d7754f7918e4d6ff49ec500d25607ce28d4b9833e454e896f849"} Nov 03 22:54:14.827931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:14.827894 4903 generic.go:356] "Generic (PLEG): container finished" podID="e3990d02-5863-49c0-a9f8-8361942d9820" containerID="94ba2dc2cc058162a3f3f2fc92a94e338abdb7481a717c8932afe17068e49736" exitCode=0 Nov 03 22:54:14.828376 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:14.827974 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" event={"ID":"e3990d02-5863-49c0-a9f8-8361942d9820","Type":"ContainerDied","Data":"94ba2dc2cc058162a3f3f2fc92a94e338abdb7481a717c8932afe17068e49736"} Nov 03 22:54:16.011938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.011907 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:16.164744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.164711 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-xks-ca\") pod \"e3990d02-5863-49c0-a9f8-8361942d9820\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " Nov 03 22:54:16.164937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.164918 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-openshift-ca\") pod \"e3990d02-5863-49c0-a9f8-8361942d9820\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " Nov 03 22:54:16.165033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.165015 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3990d02-5863-49c0-a9f8-8361942d9820-kube-api-access-xg2ng\" (UniqueName: \"kubernetes.io/projected/e3990d02-5863-49c0-a9f8-8361942d9820-kube-api-access-xg2ng\") pod \"e3990d02-5863-49c0-a9f8-8361942d9820\" (UID: \"e3990d02-5863-49c0-a9f8-8361942d9820\") " Nov 03 22:54:16.165337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.165272 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-openshift-ca" pod "e3990d02-5863-49c0-a9f8-8361942d9820" (UID: "e3990d02-5863-49c0-a9f8-8361942d9820"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:16.165626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.165605 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:16.166939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.166910 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-xks-ca" pod "e3990d02-5863-49c0-a9f8-8361942d9820" (UID: "e3990d02-5863-49c0-a9f8-8361942d9820"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:16.167063 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.167041 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3990d02-5863-49c0-a9f8-8361942d9820-kube-api-access-xg2ng" pod "e3990d02-5863-49c0-a9f8-8361942d9820" (UID: "e3990d02-5863-49c0-a9f8-8361942d9820"). InnerVolumeSpecName "kube-api-access-xg2ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:16.266855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.266820 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg2ng\" (UniqueName: \"kubernetes.io/projected/e3990d02-5863-49c0-a9f8-8361942d9820-kube-api-access-xg2ng\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:16.266855 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.266850 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3990d02-5863-49c0-a9f8-8361942d9820-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:16.846944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.846914 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" Nov 03 22:54:16.847106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.846913 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w" event={"ID":"e3990d02-5863-49c0-a9f8-8361942d9820","Type":"ContainerDied","Data":"2503a7221333d7754f7918e4d6ff49ec500d25607ce28d4b9833e454e896f849"} Nov 03 22:54:16.847106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:16.847044 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2503a7221333d7754f7918e4d6ff49ec500d25607ce28d4b9833e454e896f849" Nov 03 22:54:18.385713 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:18.385674 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:54:19.654918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.654880 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x"] Nov 03 22:54:19.928575 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.928489 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf"] Nov 03 22:54:19.929213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.929194 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e3990d02-5863-49c0-a9f8-8361942d9820" containerName="revoke-keys" Nov 03 22:54:19.929288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.929214 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3990d02-5863-49c0-a9f8-8361942d9820" containerName="revoke-keys" Nov 03 22:54:19.929409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.929397 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3990d02-5863-49c0-a9f8-8361942d9820" containerName="revoke-keys" Nov 03 22:54:19.932918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.932902 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:19.950495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.950466 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf"] Nov 03 22:54:19.952799 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.952780 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:19.955901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.955880 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:54:19.956012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.955996 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:54:19.956082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:19.956067 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:54:20.118934 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.118887 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.119114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.118940 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r495q\" (UniqueName: \"kubernetes.io/projected/f5e0400d-917c-4645-8f1c-f087ff66ae7d-kube-api-access-r495q\") pod \"maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.119324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.119257 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.220891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.220814 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.220891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.220851 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r495q\" (UniqueName: \"kubernetes.io/projected/f5e0400d-917c-4645-8f1c-f087ff66ae7d-kube-api-access-r495q\") pod \"maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.221124 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.220985 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.221468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.221446 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.223410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.223384 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.242728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.242696 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r495q\" (UniqueName: \"kubernetes.io/projected/f5e0400d-917c-4645-8f1c-f087ff66ae7d-kube-api-access-r495q\") pod \"maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.247574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.247552 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:20.552755 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.552728 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf"] Nov 03 22:54:20.554069 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:54:20.554039 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e0400d_917c_4645_8f1c_f087ff66ae7d.slice/crio-b8541298907106edca45f003c6ba96f06ae850ce33a8935ecea3c7910542734c WatchSource:0}: Error finding container b8541298907106edca45f003c6ba96f06ae850ce33a8935ecea3c7910542734c: Status 404 returned error can't find the container with id b8541298907106edca45f003c6ba96f06ae850ce33a8935ecea3c7910542734c Nov 03 22:54:20.881617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.881577 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" event={"ID":"f5e0400d-917c-4645-8f1c-f087ff66ae7d","Type":"ContainerStarted","Data":"ea844248517033d34b840a9177914116bcdb5b7fccec2179c13f2420380d47a1"} Nov 03 22:54:20.882103 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.881647 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" event={"ID":"f5e0400d-917c-4645-8f1c-f087ff66ae7d","Type":"ContainerStarted","Data":"b8541298907106edca45f003c6ba96f06ae850ce33a8935ecea3c7910542734c"} Nov 03 22:54:20.920458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:20.920404 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" podStartSLOduration=1.920388848 podStartE2EDuration="1.920388848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:20.918582646 +0000 UTC m=+2487.322523621" watchObservedRunningTime="2026-11-03 22:54:20.920388848 +0000 UTC m=+2487.324329822" Nov 03 22:54:21.891326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:21.891198 4903 generic.go:356] "Generic (PLEG): container finished" podID="f5e0400d-917c-4645-8f1c-f087ff66ae7d" containerID="ea844248517033d34b840a9177914116bcdb5b7fccec2179c13f2420380d47a1" exitCode=0 Nov 03 22:54:21.891326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:21.891273 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" event={"ID":"f5e0400d-917c-4645-8f1c-f087ff66ae7d","Type":"ContainerDied","Data":"ea844248517033d34b840a9177914116bcdb5b7fccec2179c13f2420380d47a1"} Nov 03 22:54:23.101584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.101560 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:23.171516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.171484 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-xks-ca\") pod \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " Nov 03 22:54:23.171705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.171532 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-openshift-ca\") pod \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " Nov 03 22:54:23.171705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.171604 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e0400d-917c-4645-8f1c-f087ff66ae7d-kube-api-access-r495q\" (UniqueName: \"kubernetes.io/projected/f5e0400d-917c-4645-8f1c-f087ff66ae7d-kube-api-access-r495q\") pod \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\" (UID: \"f5e0400d-917c-4645-8f1c-f087ff66ae7d\") " Nov 03 22:54:23.172176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.172000 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-openshift-ca" pod "f5e0400d-917c-4645-8f1c-f087ff66ae7d" (UID: "f5e0400d-917c-4645-8f1c-f087ff66ae7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:23.173939 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.173913 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e0400d-917c-4645-8f1c-f087ff66ae7d-kube-api-access-r495q" pod "f5e0400d-917c-4645-8f1c-f087ff66ae7d" (UID: "f5e0400d-917c-4645-8f1c-f087ff66ae7d"). InnerVolumeSpecName "kube-api-access-r495q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:23.174080 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.174058 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-xks-ca" pod "f5e0400d-917c-4645-8f1c-f087ff66ae7d" (UID: "f5e0400d-917c-4645-8f1c-f087ff66ae7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:23.273827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.273791 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:23.273827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.273819 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e0400d-917c-4645-8f1c-f087ff66ae7d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:23.273827 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.273829 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r495q\" (UniqueName: \"kubernetes.io/projected/f5e0400d-917c-4645-8f1c-f087ff66ae7d-kube-api-access-r495q\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:23.569788 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.569706 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs"] Nov 03 22:54:23.909921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.909887 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" Nov 03 22:54:23.909921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.909919 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf" event={"ID":"f5e0400d-917c-4645-8f1c-f087ff66ae7d","Type":"ContainerDied","Data":"b8541298907106edca45f003c6ba96f06ae850ce33a8935ecea3c7910542734c"} Nov 03 22:54:23.910185 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:23.909971 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8541298907106edca45f003c6ba96f06ae850ce33a8935ecea3c7910542734c" Nov 03 22:54:28.570899 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.570858 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" podUID="4644ccf9-0c96-45ef-8892-07da96b7c784" containerName="payload-processing" containerID="cri-o://ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5" gracePeriod=25 Nov 03 22:54:28.852737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.852712 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:54:28.950062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.950031 4903 generic.go:356] "Generic (PLEG): container finished" podID="4644ccf9-0c96-45ef-8892-07da96b7c784" containerID="ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5" exitCode=0 Nov 03 22:54:28.950247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.950076 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" Nov 03 22:54:28.950247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.950111 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" event={"ID":"4644ccf9-0c96-45ef-8892-07da96b7c784","Type":"ContainerDied","Data":"ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5"} Nov 03 22:54:28.950247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.950164 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs" event={"ID":"4644ccf9-0c96-45ef-8892-07da96b7c784","Type":"ContainerDied","Data":"3751261e184b0874d7a1800fffb38714dc71cdac5b8538d27c778bdca244bb4f"} Nov 03 22:54:28.950247 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.950187 4903 scope.go:122] "RemoveContainer" containerID="ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5" Nov 03 22:54:28.961279 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.961261 4903 scope.go:122] "RemoveContainer" containerID="ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5" Nov 03 22:54:28.961553 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:28.961535 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5\": container with ID starting with ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5 not found: ID does not exist" containerID="ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5" Nov 03 22:54:28.961604 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.961563 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5"} err="failed to get container status \"ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5\": rpc error: code = NotFound desc = could not find container \"ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5\": container with ID starting with ce08d54402afecd157d93fe647c5dfc76b56ffd84abc037deab5ee693b9266a5 not found: ID does not exist" Nov 03 22:54:28.963097 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.963075 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4644ccf9-0c96-45ef-8892-07da96b7c784-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644ccf9-0c96-45ef-8892-07da96b7c784-plugins-config-volume\") pod \"4644ccf9-0c96-45ef-8892-07da96b7c784\" (UID: \"4644ccf9-0c96-45ef-8892-07da96b7c784\") " Nov 03 22:54:28.963284 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.963272 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4644ccf9-0c96-45ef-8892-07da96b7c784-kube-api-access-qgbwj\" (UniqueName: \"kubernetes.io/projected/4644ccf9-0c96-45ef-8892-07da96b7c784-kube-api-access-qgbwj\") pod \"4644ccf9-0c96-45ef-8892-07da96b7c784\" (UID: \"4644ccf9-0c96-45ef-8892-07da96b7c784\") " Nov 03 22:54:28.963504 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.963476 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4644ccf9-0c96-45ef-8892-07da96b7c784-plugins-config-volume" pod "4644ccf9-0c96-45ef-8892-07da96b7c784" (UID: "4644ccf9-0c96-45ef-8892-07da96b7c784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:28.963768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.963752 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4644ccf9-0c96-45ef-8892-07da96b7c784-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:28.965335 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:28.965315 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4644ccf9-0c96-45ef-8892-07da96b7c784-kube-api-access-qgbwj" pod "4644ccf9-0c96-45ef-8892-07da96b7c784" (UID: "4644ccf9-0c96-45ef-8892-07da96b7c784"). InnerVolumeSpecName "kube-api-access-qgbwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:29.065385 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:29.065352 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgbwj\" (UniqueName: \"kubernetes.io/projected/4644ccf9-0c96-45ef-8892-07da96b7c784-kube-api-access-qgbwj\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:29.294646 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:29.294613 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs"] Nov 03 22:54:29.310929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:29.310887 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-124490-77db798586-7ghqs"] Nov 03 22:54:29.737434 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:29.737396 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4644ccf9-0c96-45ef-8892-07da96b7c784" path="/var/lib/kubelet/pods/4644ccf9-0c96-45ef-8892-07da96b7c784/volumes" Nov 03 22:54:30.382026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:30.381983 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr"] Nov 03 22:54:30.414141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:30.414109 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r"] Nov 03 22:54:34.175754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:34.175722 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk"] Nov 03 22:54:34.176156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:34.175947 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" containerID="cri-o://262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a" gracePeriod=30 Nov 03 22:54:34.413862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:34.413817 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn"] Nov 03 22:54:34.414071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:34.414045 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" containerID="cri-o://21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6" gracePeriod=30 Nov 03 22:54:34.677058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:34.676995 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" podUID="b17306d5-2b3b-4d30-9a47-14920036a2ef" containerName="main" containerID="cri-o://dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b" gracePeriod=45 Nov 03 22:54:34.819369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:34.819325 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" podUID="b17306d5-2b3b-4d30-9a47-14920036a2ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.252:8000/ready\": dial tcp 10.132.0.252:8000: connect: connection refused" Nov 03 22:54:34.973889 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:34.973863 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:54:35.013710 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.013676 4903 generic.go:356] "Generic (PLEG): container finished" podID="b17306d5-2b3b-4d30-9a47-14920036a2ef" containerID="dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b" exitCode=0 Nov 03 22:54:35.013901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.013733 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" Nov 03 22:54:35.013901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.013759 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" event={"ID":"b17306d5-2b3b-4d30-9a47-14920036a2ef","Type":"ContainerDied","Data":"dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b"} Nov 03 22:54:35.013901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.013812 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x" event={"ID":"b17306d5-2b3b-4d30-9a47-14920036a2ef","Type":"ContainerDied","Data":"a0011e156bedaff4dd81705d7850c81ce7dd2060c1ffbf7a38084bf1c307ccb7"} Nov 03 22:54:35.013901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.013834 4903 scope.go:122] "RemoveContainer" containerID="dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b" Nov 03 22:54:35.026486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.026464 4903 scope.go:122] "RemoveContainer" containerID="dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b" Nov 03 22:54:35.026979 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:35.026950 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b\": container with ID starting with dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b not found: ID does not exist" containerID="dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b" Nov 03 22:54:35.027101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.026989 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b"} err="failed to get container status \"dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b\": rpc error: code = NotFound desc = could not find container \"dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b\": container with ID starting with dd644da597f5a2927f839da5709641a496d36992d75dc1fcb1ef9a95fc1d222b not found: ID does not exist" Nov 03 22:54:35.048585 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.048552 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-home\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-home\") pod \"b17306d5-2b3b-4d30-9a47-14920036a2ef\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " Nov 03 22:54:35.048739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.048605 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17306d5-2b3b-4d30-9a47-14920036a2ef-kube-api-access-r5dpj\" (UniqueName: \"kubernetes.io/projected/b17306d5-2b3b-4d30-9a47-14920036a2ef-kube-api-access-r5dpj\") pod \"b17306d5-2b3b-4d30-9a47-14920036a2ef\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " Nov 03 22:54:35.048739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.048698 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-model-cache\") pod \"b17306d5-2b3b-4d30-9a47-14920036a2ef\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " Nov 03 22:54:35.048944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.048894 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-home" pod "b17306d5-2b3b-4d30-9a47-14920036a2ef" (UID: "b17306d5-2b3b-4d30-9a47-14920036a2ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:35.049031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.048962 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-model-cache" pod "b17306d5-2b3b-4d30-9a47-14920036a2ef" (UID: "b17306d5-2b3b-4d30-9a47-14920036a2ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:35.049031 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.048989 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-dshm\") pod \"b17306d5-2b3b-4d30-9a47-14920036a2ef\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " Nov 03 22:54:35.049165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.049151 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-tmp-dir\") pod \"b17306d5-2b3b-4d30-9a47-14920036a2ef\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " Nov 03 22:54:35.049224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.049203 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b17306d5-2b3b-4d30-9a47-14920036a2ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/b17306d5-2b3b-4d30-9a47-14920036a2ef-tls-certs\") pod \"b17306d5-2b3b-4d30-9a47-14920036a2ef\" (UID: \"b17306d5-2b3b-4d30-9a47-14920036a2ef\") " Nov 03 22:54:35.049509 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.049482 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-tmp-dir" pod "b17306d5-2b3b-4d30-9a47-14920036a2ef" (UID: "b17306d5-2b3b-4d30-9a47-14920036a2ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:35.049920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.049883 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:35.049920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.049909 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:35.050102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.049926 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:35.051987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.051956 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-dshm" pod "b17306d5-2b3b-4d30-9a47-14920036a2ef" (UID: "b17306d5-2b3b-4d30-9a47-14920036a2ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:35.052098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.052023 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17306d5-2b3b-4d30-9a47-14920036a2ef-tls-certs" pod "b17306d5-2b3b-4d30-9a47-14920036a2ef" (UID: "b17306d5-2b3b-4d30-9a47-14920036a2ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:35.054452 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.054422 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17306d5-2b3b-4d30-9a47-14920036a2ef-kube-api-access-r5dpj" pod "b17306d5-2b3b-4d30-9a47-14920036a2ef" (UID: "b17306d5-2b3b-4d30-9a47-14920036a2ef"). InnerVolumeSpecName "kube-api-access-r5dpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:35.151197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.151166 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5dpj\" (UniqueName: \"kubernetes.io/projected/b17306d5-2b3b-4d30-9a47-14920036a2ef-kube-api-access-r5dpj\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:35.151197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.151194 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17306d5-2b3b-4d30-9a47-14920036a2ef-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:35.151197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.151205 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17306d5-2b3b-4d30-9a47-14920036a2ef-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:35.260668 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.260582 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z"] Nov 03 22:54:35.261863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.261834 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f5e0400d-917c-4645-8f1c-f087ff66ae7d" containerName="revoke-keys" Nov 03 22:54:35.261863 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.261860 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e0400d-917c-4645-8f1c-f087ff66ae7d" containerName="revoke-keys" Nov 03 22:54:35.262066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.261888 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b17306d5-2b3b-4d30-9a47-14920036a2ef" containerName="main" Nov 03 22:54:35.262066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.261896 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17306d5-2b3b-4d30-9a47-14920036a2ef" containerName="main" Nov 03 22:54:35.262066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.261940 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4644ccf9-0c96-45ef-8892-07da96b7c784" containerName="payload-processing" Nov 03 22:54:35.262066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.261949 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4644ccf9-0c96-45ef-8892-07da96b7c784" containerName="payload-processing" Nov 03 22:54:35.262278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.262132 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4644ccf9-0c96-45ef-8892-07da96b7c784" containerName="payload-processing" Nov 03 22:54:35.262278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.262153 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e0400d-917c-4645-8f1c-f087ff66ae7d" containerName="revoke-keys" Nov 03 22:54:35.262278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.262163 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17306d5-2b3b-4d30-9a47-14920036a2ef" containerName="main" Nov 03 22:54:35.269456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.268097 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.291045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.287620 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-26f8ba\"/\"sub-test-model-26f8ba-kserve-self-signed-certs\"" Nov 03 22:54:35.291045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.288272 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-26f8ba\"/\"kube-root-ca.crt\"" Nov 03 22:54:35.293512 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.293486 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z"] Nov 03 22:54:35.297931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.297905 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-26f8ba\"/\"openshift-service-ca.crt\"" Nov 03 22:54:35.297931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.297909 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-26f8ba\"/\"default-dockercfg-cml95\"" Nov 03 22:54:35.353562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.353534 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-home\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.353740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.353595 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-dshm\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.353740 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.353707 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-model-cache\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.353846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.353750 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-tmp-dir\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.353992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.353966 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e7a893-2eba-47df-835c-565ebb6c1436-tls-certs\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.354062 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.354048 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lznc\" (UniqueName: \"kubernetes.io/projected/99e7a893-2eba-47df-835c-565ebb6c1436-kube-api-access-2lznc\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.362989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.362965 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x"] Nov 03 22:54:35.367705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.367686 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-963936/test-model-963936-kserve-5fdcd9bc5f-cqj4x"] Nov 03 22:54:35.382914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.382881 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" podUID="7a0c63e2-19da-49d8-84b7-f826385c2d68" containerName="payload-processing" containerID="cri-o://39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73" gracePeriod=25 Nov 03 22:54:35.415051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.415011 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" podUID="cea3c3cb-4508-409d-afb4-a03249c5702b" containerName="payload-pre-processing" containerID="cri-o://6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c" gracePeriod=25 Nov 03 22:54:35.454929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.454899 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-model-cache\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.455122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.454947 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-tmp-dir\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.455122 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.455084 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e7a893-2eba-47df-835c-565ebb6c1436-tls-certs\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.455220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.455141 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lznc\" (UniqueName: \"kubernetes.io/projected/99e7a893-2eba-47df-835c-565ebb6c1436-kube-api-access-2lznc\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.455220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.455163 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-home\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.455346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.455217 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-dshm\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.455424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.455367 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-model-cache\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.455424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.455416 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-tmp-dir\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.455837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.455814 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-home\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.457426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.457406 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-dshm\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.457810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.457790 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e7a893-2eba-47df-835c-565ebb6c1436-tls-certs\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.483410 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.483384 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lznc\" (UniqueName: \"kubernetes.io/projected/99e7a893-2eba-47df-835c-565ebb6c1436-kube-api-access-2lznc\") pod \"sub-test-model-26f8ba-kserve-96c894f67-wzh5z\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.599499 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.599079 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:35.770111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.770082 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17306d5-2b3b-4d30-9a47-14920036a2ef" path="/var/lib/kubelet/pods/b17306d5-2b3b-4d30-9a47-14920036a2ef/volumes" Nov 03 22:54:35.921803 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:35.921768 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z"] Nov 03 22:54:35.922769 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:54:35.922739 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99e7a893_2eba_47df_835c_565ebb6c1436.slice/crio-a82ebb80d4e77f2b7e11c3ba3e648442efa37371c390b31244a5f4862139b4b9 WatchSource:0}: Error finding container a82ebb80d4e77f2b7e11c3ba3e648442efa37371c390b31244a5f4862139b4b9: Status 404 returned error can't find the container with id a82ebb80d4e77f2b7e11c3ba3e648442efa37371c390b31244a5f4862139b4b9 Nov 03 22:54:36.025679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:36.025636 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" event={"ID":"99e7a893-2eba-47df-835c-565ebb6c1436","Type":"ContainerStarted","Data":"5823963ad9d046b90701b24b9f459aaa282d1f2864ba55efb098c97b3459cc72"} Nov 03 22:54:36.025857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:36.025689 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" event={"ID":"99e7a893-2eba-47df-835c-565ebb6c1436","Type":"ContainerStarted","Data":"a82ebb80d4e77f2b7e11c3ba3e648442efa37371c390b31244a5f4862139b4b9"} Nov 03 22:54:36.064116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:36.064056 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" podStartSLOduration=1.06404062 podStartE2EDuration="1.06404062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:36.063596132 +0000 UTC m=+2502.467537107" watchObservedRunningTime="2026-11-03 22:54:36.06404062 +0000 UTC m=+2502.467981596" Nov 03 22:54:36.599529 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:36.599494 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:37.683834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:37.683792 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" podUID="cea3c3cb-4508-409d-afb4-a03249c5702b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:54:37.684258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:37.683853 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" podUID="7a0c63e2-19da-49d8-84b7-f826385c2d68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:54:37.709631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:37.709595 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf"] Nov 03 22:54:37.714374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:37.714350 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb1fa5-v2cvf"] Nov 03 22:54:37.737140 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:37.737108 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e0400d-917c-4645-8f1c-f087ff66ae7d" path="/var/lib/kubelet/pods/f5e0400d-917c-4645-8f1c-f087ff66ae7d/volumes" Nov 03 22:54:39.546818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.546792 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:54:39.644285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.644253 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-envoy\") pod \"acf9ce64-ce17-4f22-9a81-b038a7987369\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " Nov 03 22:54:39.644509 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.644438 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-certs\") pod \"acf9ce64-ce17-4f22-9a81-b038a7987369\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " Nov 03 22:54:39.644589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.644536 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-kube-api-access-4nzsn\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-kube-api-access-4nzsn\") pod \"acf9ce64-ce17-4f22-9a81-b038a7987369\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " Nov 03 22:54:39.644589 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.644570 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-socket\") pod \"acf9ce64-ce17-4f22-9a81-b038a7987369\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " Nov 03 22:54:39.644715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.644664 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-podinfo\") pod \"acf9ce64-ce17-4f22-9a81-b038a7987369\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " Nov 03 22:54:39.644715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.644705 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf9ce64-ce17-4f22-9a81-b038a7987369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf9ce64-ce17-4f22-9a81-b038a7987369-istiod-ca-cert\") pod \"acf9ce64-ce17-4f22-9a81-b038a7987369\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " Nov 03 22:54:39.644837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.644818 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-credential-socket\") pod \"acf9ce64-ce17-4f22-9a81-b038a7987369\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " Nov 03 22:54:39.644893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.644848 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-token\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-token\") pod \"acf9ce64-ce17-4f22-9a81-b038a7987369\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " Nov 03 22:54:39.644955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.644903 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-data\") pod \"acf9ce64-ce17-4f22-9a81-b038a7987369\" (UID: \"acf9ce64-ce17-4f22-9a81-b038a7987369\") " Nov 03 22:54:39.645797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.645243 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf9ce64-ce17-4f22-9a81-b038a7987369-istiod-ca-cert" pod "acf9ce64-ce17-4f22-9a81-b038a7987369" (UID: "acf9ce64-ce17-4f22-9a81-b038a7987369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:39.645797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.645515 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-credential-socket" pod "acf9ce64-ce17-4f22-9a81-b038a7987369" (UID: "acf9ce64-ce17-4f22-9a81-b038a7987369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.645797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.645527 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-data" pod "acf9ce64-ce17-4f22-9a81-b038a7987369" (UID: "acf9ce64-ce17-4f22-9a81-b038a7987369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.645797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.645733 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acf9ce64-ce17-4f22-9a81-b038a7987369-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.645797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.645758 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.645797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.645773 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.646317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.646262 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-socket" pod "acf9ce64-ce17-4f22-9a81-b038a7987369" (UID: "acf9ce64-ce17-4f22-9a81-b038a7987369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.646495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.646465 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-certs" pod "acf9ce64-ce17-4f22-9a81-b038a7987369" (UID: "acf9ce64-ce17-4f22-9a81-b038a7987369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.648440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.648413 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-token" pod "acf9ce64-ce17-4f22-9a81-b038a7987369" (UID: "acf9ce64-ce17-4f22-9a81-b038a7987369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:39.649348 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.649320 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-podinfo" pod "acf9ce64-ce17-4f22-9a81-b038a7987369" (UID: "acf9ce64-ce17-4f22-9a81-b038a7987369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:39.649442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.649424 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-envoy" pod "acf9ce64-ce17-4f22-9a81-b038a7987369" (UID: "acf9ce64-ce17-4f22-9a81-b038a7987369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.650328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.650280 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-kube-api-access-4nzsn" pod "acf9ce64-ce17-4f22-9a81-b038a7987369" (UID: "acf9ce64-ce17-4f22-9a81-b038a7987369"). InnerVolumeSpecName "kube-api-access-4nzsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:39.685101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.685079 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:54:39.746795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.746772 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.746795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.746795 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.746961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.746805 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.746961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.746814 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nzsn\" (UniqueName: \"kubernetes.io/projected/acf9ce64-ce17-4f22-9a81-b038a7987369-kube-api-access-4nzsn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.746961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.746824 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acf9ce64-ce17-4f22-9a81-b038a7987369-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.746961 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.746835 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acf9ce64-ce17-4f22-9a81-b038a7987369-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.847763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.847723 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-credential-socket\") pod \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " Nov 03 22:54:39.847981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.847785 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istiod-ca-cert\") pod \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " Nov 03 22:54:39.847981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.847807 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-data\") pod \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " Nov 03 22:54:39.848111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.847982 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-token\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-token\") pod \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " Nov 03 22:54:39.848111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848035 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-credential-socket" pod "849e7c9f-8596-41a4-b9ed-e7472cecadbb" (UID: "849e7c9f-8596-41a4-b9ed-e7472cecadbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.848111 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848043 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-certs\") pod \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " Nov 03 22:54:39.848287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848129 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-data" pod "849e7c9f-8596-41a4-b9ed-e7472cecadbb" (UID: "849e7c9f-8596-41a4-b9ed-e7472cecadbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.848287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848188 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-kube-api-access-g2xgr\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-kube-api-access-g2xgr\") pod \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " Nov 03 22:54:39.848451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848289 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-certs" pod "849e7c9f-8596-41a4-b9ed-e7472cecadbb" (UID: "849e7c9f-8596-41a4-b9ed-e7472cecadbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.848451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848288 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istiod-ca-cert" pod "849e7c9f-8596-41a4-b9ed-e7472cecadbb" (UID: "849e7c9f-8596-41a4-b9ed-e7472cecadbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:39.848451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848337 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-socket\") pod \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " Nov 03 22:54:39.848451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848427 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-envoy\") pod \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " Nov 03 22:54:39.848669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848514 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-socket" pod "849e7c9f-8596-41a4-b9ed-e7472cecadbb" (UID: "849e7c9f-8596-41a4-b9ed-e7472cecadbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.848669 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.848572 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-podinfo\") pod \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\" (UID: \"849e7c9f-8596-41a4-b9ed-e7472cecadbb\") " Nov 03 22:54:39.849143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.849124 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.849238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.849145 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.849238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.849158 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.849238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.849170 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.849238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.849183 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.850409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.850387 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-envoy" pod "849e7c9f-8596-41a4-b9ed-e7472cecadbb" (UID: "849e7c9f-8596-41a4-b9ed-e7472cecadbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:54:39.850477 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.850407 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-kube-api-access-g2xgr" pod "849e7c9f-8596-41a4-b9ed-e7472cecadbb" (UID: "849e7c9f-8596-41a4-b9ed-e7472cecadbb"). InnerVolumeSpecName "kube-api-access-g2xgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:39.850677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.850664 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-token" pod "849e7c9f-8596-41a4-b9ed-e7472cecadbb" (UID: "849e7c9f-8596-41a4-b9ed-e7472cecadbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:39.850808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.850790 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-podinfo" pod "849e7c9f-8596-41a4-b9ed-e7472cecadbb" (UID: "849e7c9f-8596-41a4-b9ed-e7472cecadbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:54:39.950949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.950917 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.950949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.950945 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2xgr\" (UniqueName: \"kubernetes.io/projected/849e7c9f-8596-41a4-b9ed-e7472cecadbb-kube-api-access-g2xgr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.950949 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.950955 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:39.951199 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:39.950963 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849e7c9f-8596-41a4-b9ed-e7472cecadbb-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:40.060915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.060836 4903 generic.go:356] "Generic (PLEG): container finished" podID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerID="21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6" exitCode=0 Nov 03 22:54:40.060915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.060895 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" Nov 03 22:54:40.061159 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.060917 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" event={"ID":"849e7c9f-8596-41a4-b9ed-e7472cecadbb","Type":"ContainerDied","Data":"21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6"} Nov 03 22:54:40.061159 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.060975 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn" event={"ID":"849e7c9f-8596-41a4-b9ed-e7472cecadbb","Type":"ContainerDied","Data":"766e707f049ee9c320a3fae86e8b3f8735a1cb27c5025b94d64528aec6849f11"} Nov 03 22:54:40.061159 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.061019 4903 scope.go:122] "RemoveContainer" containerID="21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6" Nov 03 22:54:40.062960 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.062938 4903 generic.go:356] "Generic (PLEG): container finished" podID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerID="262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a" exitCode=0 Nov 03 22:54:40.063104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.062996 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" Nov 03 22:54:40.063104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.063023 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" event={"ID":"acf9ce64-ce17-4f22-9a81-b038a7987369","Type":"ContainerDied","Data":"262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a"} Nov 03 22:54:40.063104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.063063 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk" event={"ID":"acf9ce64-ce17-4f22-9a81-b038a7987369","Type":"ContainerDied","Data":"d02f8860cb8a8130e67abb8a9701f4b31b06c1a80a0369edaba027e099a82d2b"} Nov 03 22:54:40.073107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.073092 4903 scope.go:122] "RemoveContainer" containerID="9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5" Nov 03 22:54:40.083331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.083284 4903 scope.go:122] "RemoveContainer" containerID="21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6" Nov 03 22:54:40.083625 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:40.083607 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6\": container with ID starting with 21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6 not found: ID does not exist" containerID="21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6" Nov 03 22:54:40.083678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.083633 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6"} err="failed to get container status \"21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6\": rpc error: code = NotFound desc = could not find container \"21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6\": container with ID starting with 21c7e8a407f17a5e56214ca1f8a5fec5d7e0212acaf9b53557af257ac45536d6 not found: ID does not exist" Nov 03 22:54:40.083678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.083652 4903 scope.go:122] "RemoveContainer" containerID="9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5" Nov 03 22:54:40.083897 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:40.083875 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5\": container with ID starting with 9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5 not found: ID does not exist" containerID="9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5" Nov 03 22:54:40.083974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.083907 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5"} err="failed to get container status \"9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5\": rpc error: code = NotFound desc = could not find container \"9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5\": container with ID starting with 9495c7151f0b1d3e39f4d77b6a78da18cd42b495d06cafd79a1e8ca277350ff5 not found: ID does not exist" Nov 03 22:54:40.083974 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.083930 4903 scope.go:122] "RemoveContainer" containerID="262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a" Nov 03 22:54:40.093456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.093431 4903 scope.go:122] "RemoveContainer" containerID="70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955" Nov 03 22:54:40.103557 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.103537 4903 scope.go:122] "RemoveContainer" containerID="262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a" Nov 03 22:54:40.103875 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:40.103843 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a\": container with ID starting with 262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a not found: ID does not exist" containerID="262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a" Nov 03 22:54:40.103964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.103881 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a"} err="failed to get container status \"262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a\": rpc error: code = NotFound desc = could not find container \"262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a\": container with ID starting with 262589b8cdf60e29f6e42e064f30f39926d96005e4f290662210cd2308e48d9a not found: ID does not exist" Nov 03 22:54:40.103964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.103899 4903 scope.go:122] "RemoveContainer" containerID="70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955" Nov 03 22:54:40.104147 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:40.104129 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955\": container with ID starting with 70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955 not found: ID does not exist" containerID="70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955" Nov 03 22:54:40.104224 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.104152 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955"} err="failed to get container status \"70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955\": rpc error: code = NotFound desc = could not find container \"70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955\": container with ID starting with 70c5d2d6f2b274a331f9311de369aea1fc7ad65ba03de2fbe000a0d7a5abf955 not found: ID does not exist" Nov 03 22:54:40.111134 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.111107 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk"] Nov 03 22:54:40.121223 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.121197 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2d619f-openshift-default-85b84758df-nnfpk"] Nov 03 22:54:40.159625 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.159590 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn"] Nov 03 22:54:40.166398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:40.166372 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-434ce3-openshift-default-58c7774759-s5wxn"] Nov 03 22:54:41.739479 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:41.739440 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" path="/var/lib/kubelet/pods/849e7c9f-8596-41a4-b9ed-e7472cecadbb/volumes" Nov 03 22:54:41.740143 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:41.740125 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" path="/var/lib/kubelet/pods/acf9ce64-ce17-4f22-9a81-b038a7987369/volumes" Nov 03 22:54:41.825249 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:41.825213 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:54:45.599468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.599427 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:45.622967 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.622935 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:45.971862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.971829 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk"] Nov 03 22:54:45.972539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972524 4903 cpu_manager.go:414] "containerMap: removing container" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" Nov 03 22:54:45.972605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972540 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" Nov 03 22:54:45.972605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972563 4903 cpu_manager.go:414] "containerMap: removing container" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" Nov 03 22:54:45.972605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972568 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" Nov 03 22:54:45.972605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972580 4903 cpu_manager.go:414] "containerMap: removing container" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" Nov 03 22:54:45.972605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972585 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" Nov 03 22:54:45.972605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972594 4903 cpu_manager.go:414] "containerMap: removing container" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" Nov 03 22:54:45.972605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972599 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" Nov 03 22:54:45.972854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972729 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" Nov 03 22:54:45.972854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972743 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="849e7c9f-8596-41a4-b9ed-e7472cecadbb" containerName="istio-proxy" Nov 03 22:54:45.972854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972750 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" Nov 03 22:54:45.972854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.972756 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf9ce64-ce17-4f22-9a81-b038a7987369" containerName="istio-proxy" Nov 03 22:54:45.978272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.978255 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:45.992877 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.992853 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:45.994278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.994254 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk"] Nov 03 22:54:45.999594 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.999573 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:54:45.999679 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:45.999661 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:54:46.001500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.001481 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:54:46.116045 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.116018 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6"] Nov 03 22:54:46.125691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.125669 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.126137 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.126117 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:54:46.138287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.138263 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6"] Nov 03 22:54:46.145707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.145675 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9h7v\" (UniqueName: \"kubernetes.io/projected/2b072383-4e99-4952-a66a-1eeb8e0bffbc-kube-api-access-f9h7v\") pod \"maas-api-revoke-sub-81360fe2-9jldk\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.145862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.145744 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81360fe2-9jldk\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.145934 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.145908 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81360fe2-9jldk\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.247552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.247463 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r86tf\" (UniqueName: \"kubernetes.io/projected/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-kube-api-access-r86tf\") pod \"maas-api-revoke-keys-e2e-inf-b-963936-dc6f6\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.247552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.247524 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81360fe2-9jldk\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.247552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.247551 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963936-dc6f6\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.247859 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.247794 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963936-dc6f6\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.247924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.247902 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9h7v\" (UniqueName: \"kubernetes.io/projected/2b072383-4e99-4952-a66a-1eeb8e0bffbc-kube-api-access-f9h7v\") pod \"maas-api-revoke-sub-81360fe2-9jldk\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.247982 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.247947 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81360fe2-9jldk\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.248703 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.248677 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81360fe2-9jldk\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.253328 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.249882 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81360fe2-9jldk\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.269851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.269815 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9h7v\" (UniqueName: \"kubernetes.io/projected/2b072383-4e99-4952-a66a-1eeb8e0bffbc-kube-api-access-f9h7v\") pod \"maas-api-revoke-sub-81360fe2-9jldk\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.293364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.293341 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:46.349736 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.349699 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963936-dc6f6\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.349925 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.349904 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86tf\" (UniqueName: \"kubernetes.io/projected/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-kube-api-access-r86tf\") pod \"maas-api-revoke-keys-e2e-inf-b-963936-dc6f6\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.350000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.349967 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963936-dc6f6\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.351343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.351175 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963936-dc6f6\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.358901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.358878 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-963936-dc6f6\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.378697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.378480 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86tf\" (UniqueName: \"kubernetes.io/projected/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-kube-api-access-r86tf\") pod \"maas-api-revoke-keys-e2e-inf-b-963936-dc6f6\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.441533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.441503 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:46.625109 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.625081 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk"] Nov 03 22:54:46.752305 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:46.752220 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6"] Nov 03 22:54:46.752430 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:54:46.752272 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe12e9d_54ea_42b1_ba23_28edfd8826e7.slice/crio-5c1e572d3553a1995c59bf0da54686fa116b4ae2f55eff4765b03a548996be0b WatchSource:0}: Error finding container 5c1e572d3553a1995c59bf0da54686fa116b4ae2f55eff4765b03a548996be0b: Status 404 returned error can't find the container with id 5c1e572d3553a1995c59bf0da54686fa116b4ae2f55eff4765b03a548996be0b Nov 03 22:54:47.124749 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:47.124711 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" event={"ID":"3fe12e9d-54ea-42b1-ba23-28edfd8826e7","Type":"ContainerStarted","Data":"5c1e572d3553a1995c59bf0da54686fa116b4ae2f55eff4765b03a548996be0b"} Nov 03 22:54:47.126486 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:47.126455 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" event={"ID":"2b072383-4e99-4952-a66a-1eeb8e0bffbc","Type":"ContainerStarted","Data":"eb5a53a399f0103f9366ff38df847596ff43c03913362628d8745cb96d44666b"} Nov 03 22:54:47.126619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:47.126506 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" event={"ID":"2b072383-4e99-4952-a66a-1eeb8e0bffbc","Type":"ContainerStarted","Data":"57000dcb6fc1cb8cb1fbc099e466885bf6ed0d65e6b6dddc95bedb1642620a74"} Nov 03 22:54:47.166219 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:47.166167 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" podStartSLOduration=2.166150678 podStartE2EDuration="2.166150678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:47.162716116 +0000 UTC m=+2513.566657091" watchObservedRunningTime="2026-11-03 22:54:47.166150678 +0000 UTC m=+2513.570091649" Nov 03 22:54:47.683407 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:47.683362 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" podUID="cea3c3cb-4508-409d-afb4-a03249c5702b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:54:47.683793 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:47.683769 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" podUID="7a0c63e2-19da-49d8-84b7-f826385c2d68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:54:48.136518 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:48.136487 4903 generic.go:356] "Generic (PLEG): container finished" podID="3fe12e9d-54ea-42b1-ba23-28edfd8826e7" containerID="ec82397e7e647dd7115dd05ea7d0983b35bad79a2a9c4e98d62ab97730310280" exitCode=0 Nov 03 22:54:48.136689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:48.136566 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" event={"ID":"3fe12e9d-54ea-42b1-ba23-28edfd8826e7","Type":"ContainerDied","Data":"ec82397e7e647dd7115dd05ea7d0983b35bad79a2a9c4e98d62ab97730310280"} Nov 03 22:54:48.138336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:48.138285 4903 generic.go:356] "Generic (PLEG): container finished" podID="2b072383-4e99-4952-a66a-1eeb8e0bffbc" containerID="eb5a53a399f0103f9366ff38df847596ff43c03913362628d8745cb96d44666b" exitCode=0 Nov 03 22:54:48.138473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:48.138354 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" event={"ID":"2b072383-4e99-4952-a66a-1eeb8e0bffbc","Type":"ContainerDied","Data":"eb5a53a399f0103f9366ff38df847596ff43c03913362628d8745cb96d44666b"} Nov 03 22:54:48.393368 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:48.393252 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:54:49.355087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.355060 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:49.358290 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.358272 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:49.392849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.392822 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-xks-ca\") pod \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " Nov 03 22:54:49.393040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.392893 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b072383-4e99-4952-a66a-1eeb8e0bffbc-kube-api-access-f9h7v\" (UniqueName: \"kubernetes.io/projected/2b072383-4e99-4952-a66a-1eeb8e0bffbc-kube-api-access-f9h7v\") pod \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " Nov 03 22:54:49.394120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.393260 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-openshift-ca\") pod \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " Nov 03 22:54:49.394120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.393386 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-openshift-ca\") pod \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\" (UID: \"2b072383-4e99-4952-a66a-1eeb8e0bffbc\") " Nov 03 22:54:49.394120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.393462 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-xks-ca\") pod \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " Nov 03 22:54:49.394120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.393607 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-kube-api-access-r86tf\" (UniqueName: \"kubernetes.io/projected/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-kube-api-access-r86tf\") pod \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\" (UID: \"3fe12e9d-54ea-42b1-ba23-28edfd8826e7\") " Nov 03 22:54:49.394120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.393809 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-openshift-ca" pod "2b072383-4e99-4952-a66a-1eeb8e0bffbc" (UID: "2b072383-4e99-4952-a66a-1eeb8e0bffbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:49.394120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.394086 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-openshift-ca" pod "3fe12e9d-54ea-42b1-ba23-28edfd8826e7" (UID: "3fe12e9d-54ea-42b1-ba23-28edfd8826e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:49.394677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.394322 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:49.394677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.394344 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:49.408726 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.408694 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-xks-ca" pod "2b072383-4e99-4952-a66a-1eeb8e0bffbc" (UID: "2b072383-4e99-4952-a66a-1eeb8e0bffbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:49.411619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.411580 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-kube-api-access-r86tf" pod "3fe12e9d-54ea-42b1-ba23-28edfd8826e7" (UID: "3fe12e9d-54ea-42b1-ba23-28edfd8826e7"). InnerVolumeSpecName "kube-api-access-r86tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:49.440830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.440800 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b072383-4e99-4952-a66a-1eeb8e0bffbc-kube-api-access-f9h7v" pod "2b072383-4e99-4952-a66a-1eeb8e0bffbc" (UID: "2b072383-4e99-4952-a66a-1eeb8e0bffbc"). InnerVolumeSpecName "kube-api-access-f9h7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:49.447334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.447306 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-xks-ca" pod "3fe12e9d-54ea-42b1-ba23-28edfd8826e7" (UID: "3fe12e9d-54ea-42b1-ba23-28edfd8826e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:49.496081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.496042 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r86tf\" (UniqueName: \"kubernetes.io/projected/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-kube-api-access-r86tf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:49.496081 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.496085 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b072383-4e99-4952-a66a-1eeb8e0bffbc-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:49.496288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.496102 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9h7v\" (UniqueName: \"kubernetes.io/projected/2b072383-4e99-4952-a66a-1eeb8e0bffbc-kube-api-access-f9h7v\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:49.496288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:49.496117 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe12e9d-54ea-42b1-ba23-28edfd8826e7-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:50.077107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:50.077018 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w"] Nov 03 22:54:50.080384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:50.080360 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-124490-27m9w"] Nov 03 22:54:50.181048 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:50.181006 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" Nov 03 22:54:50.181263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:50.181015 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6" event={"ID":"3fe12e9d-54ea-42b1-ba23-28edfd8826e7","Type":"ContainerDied","Data":"5c1e572d3553a1995c59bf0da54686fa116b4ae2f55eff4765b03a548996be0b"} Nov 03 22:54:50.181263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:50.181139 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c1e572d3553a1995c59bf0da54686fa116b4ae2f55eff4765b03a548996be0b" Nov 03 22:54:50.182846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:50.182827 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" Nov 03 22:54:50.182992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:50.182964 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk" event={"ID":"2b072383-4e99-4952-a66a-1eeb8e0bffbc","Type":"ContainerDied","Data":"57000dcb6fc1cb8cb1fbc099e466885bf6ed0d65e6b6dddc95bedb1642620a74"} Nov 03 22:54:50.183068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:50.183002 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57000dcb6fc1cb8cb1fbc099e466885bf6ed0d65e6b6dddc95bedb1642620a74" Nov 03 22:54:51.734966 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:51.734933 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3990d02-5863-49c0-a9f8-8361942d9820" path="/var/lib/kubelet/pods/e3990d02-5863-49c0-a9f8-8361942d9820/volumes" Nov 03 22:54:55.732903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.732876 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:54:55.737804 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.737782 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:54:55.786281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.786250 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0c63e2-19da-49d8-84b7-f826385c2d68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0c63e2-19da-49d8-84b7-f826385c2d68-plugins-config-volume\") pod \"7a0c63e2-19da-49d8-84b7-f826385c2d68\" (UID: \"7a0c63e2-19da-49d8-84b7-f826385c2d68\") " Nov 03 22:54:55.786489 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.786358 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea3c3cb-4508-409d-afb4-a03249c5702b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea3c3cb-4508-409d-afb4-a03249c5702b-plugins-config-volume\") pod \"cea3c3cb-4508-409d-afb4-a03249c5702b\" (UID: \"cea3c3cb-4508-409d-afb4-a03249c5702b\") " Nov 03 22:54:55.786559 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.786490 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0c63e2-19da-49d8-84b7-f826385c2d68-kube-api-access-sqf7l\" (UniqueName: \"kubernetes.io/projected/7a0c63e2-19da-49d8-84b7-f826385c2d68-kube-api-access-sqf7l\") pod \"7a0c63e2-19da-49d8-84b7-f826385c2d68\" (UID: \"7a0c63e2-19da-49d8-84b7-f826385c2d68\") " Nov 03 22:54:55.786677 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.786662 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea3c3cb-4508-409d-afb4-a03249c5702b-kube-api-access-tjctv\" (UniqueName: \"kubernetes.io/projected/cea3c3cb-4508-409d-afb4-a03249c5702b-kube-api-access-tjctv\") pod \"cea3c3cb-4508-409d-afb4-a03249c5702b\" (UID: \"cea3c3cb-4508-409d-afb4-a03249c5702b\") " Nov 03 22:54:55.786768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.786736 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0c63e2-19da-49d8-84b7-f826385c2d68-plugins-config-volume" pod "7a0c63e2-19da-49d8-84b7-f826385c2d68" (UID: "7a0c63e2-19da-49d8-84b7-f826385c2d68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:55.786829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.786795 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea3c3cb-4508-409d-afb4-a03249c5702b-plugins-config-volume" pod "cea3c3cb-4508-409d-afb4-a03249c5702b" (UID: "cea3c3cb-4508-409d-afb4-a03249c5702b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:55.787209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.787187 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0c63e2-19da-49d8-84b7-f826385c2d68-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:55.787372 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.787355 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea3c3cb-4508-409d-afb4-a03249c5702b-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:55.789433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.789407 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0c63e2-19da-49d8-84b7-f826385c2d68-kube-api-access-sqf7l" pod "7a0c63e2-19da-49d8-84b7-f826385c2d68" (UID: "7a0c63e2-19da-49d8-84b7-f826385c2d68"). InnerVolumeSpecName "kube-api-access-sqf7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:55.790832 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.790784 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea3c3cb-4508-409d-afb4-a03249c5702b-kube-api-access-tjctv" pod "cea3c3cb-4508-409d-afb4-a03249c5702b" (UID: "cea3c3cb-4508-409d-afb4-a03249c5702b"). InnerVolumeSpecName "kube-api-access-tjctv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:55.888918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.888881 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqf7l\" (UniqueName: \"kubernetes.io/projected/7a0c63e2-19da-49d8-84b7-f826385c2d68-kube-api-access-sqf7l\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:55.888918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:55.888912 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjctv\" (UniqueName: \"kubernetes.io/projected/cea3c3cb-4508-409d-afb4-a03249c5702b-kube-api-access-tjctv\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:56.231186 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.231153 4903 generic.go:356] "Generic (PLEG): container finished" podID="cea3c3cb-4508-409d-afb4-a03249c5702b" containerID="6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c" exitCode=0 Nov 03 22:54:56.231424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.231240 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" Nov 03 22:54:56.231424 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.231249 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" event={"ID":"cea3c3cb-4508-409d-afb4-a03249c5702b","Type":"ContainerDied","Data":"6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c"} Nov 03 22:54:56.231542 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.231434 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r" event={"ID":"cea3c3cb-4508-409d-afb4-a03249c5702b","Type":"ContainerDied","Data":"718fd7abdd81a191ac0d0cb66a943d580976ab9816c11f7621ec3a40f07c36af"} Nov 03 22:54:56.231542 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.231462 4903 scope.go:122] "RemoveContainer" containerID="6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c" Nov 03 22:54:56.232862 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.232842 4903 generic.go:356] "Generic (PLEG): container finished" podID="7a0c63e2-19da-49d8-84b7-f826385c2d68" containerID="39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73" exitCode=0 Nov 03 22:54:56.232967 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.232897 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" Nov 03 22:54:56.232967 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.232908 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" event={"ID":"7a0c63e2-19da-49d8-84b7-f826385c2d68","Type":"ContainerDied","Data":"39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73"} Nov 03 22:54:56.232967 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.232954 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr" event={"ID":"7a0c63e2-19da-49d8-84b7-f826385c2d68","Type":"ContainerDied","Data":"d10f83f23c4e01979d622502ebc99e8a71ae42601c84c90ba9000039f0733db9"} Nov 03 22:54:56.243532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.243019 4903 scope.go:122] "RemoveContainer" containerID="6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c" Nov 03 22:54:56.243532 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:56.243281 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c\": container with ID starting with 6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c not found: ID does not exist" containerID="6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c" Nov 03 22:54:56.243532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.243364 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c"} err="failed to get container status \"6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c\": rpc error: code = NotFound desc = could not find container \"6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c\": container with ID starting with 6d45089635ace3d27ac40e828832a3de4e2b94c8bb7eb0878c01a568ce594c9c not found: ID does not exist" Nov 03 22:54:56.243532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.243386 4903 scope.go:122] "RemoveContainer" containerID="39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73" Nov 03 22:54:56.261791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.261771 4903 scope.go:122] "RemoveContainer" containerID="39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73" Nov 03 22:54:56.262091 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:54:56.262062 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73\": container with ID starting with 39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73 not found: ID does not exist" containerID="39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73" Nov 03 22:54:56.262175 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.262098 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73"} err="failed to get container status \"39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73\": rpc error: code = NotFound desc = could not find container \"39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73\": container with ID starting with 39ea76af42da8c25ba761bf39b0eadf228fee2627e66d4a9d60852d740f40d73 not found: ID does not exist" Nov 03 22:54:56.291848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.291820 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr"] Nov 03 22:54:56.295473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.295448 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb1fa5-6694d587bb-tdvvr"] Nov 03 22:54:56.324715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.324690 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r"] Nov 03 22:54:56.328944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.328918 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bb1fa5-7948594b67-45z4r"] Nov 03 22:54:56.400851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.400818 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c"] Nov 03 22:54:56.401873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.401840 4903 cpu_manager.go:414] "containerMap: removing container" podUID="cea3c3cb-4508-409d-afb4-a03249c5702b" containerName="payload-pre-processing" Nov 03 22:54:56.401873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.401864 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea3c3cb-4508-409d-afb4-a03249c5702b" containerName="payload-pre-processing" Nov 03 22:54:56.402077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.401880 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7a0c63e2-19da-49d8-84b7-f826385c2d68" containerName="payload-processing" Nov 03 22:54:56.402077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.401889 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0c63e2-19da-49d8-84b7-f826385c2d68" containerName="payload-processing" Nov 03 22:54:56.402077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.401919 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2b072383-4e99-4952-a66a-1eeb8e0bffbc" containerName="revoke-keys" Nov 03 22:54:56.402077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.401926 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b072383-4e99-4952-a66a-1eeb8e0bffbc" containerName="revoke-keys" Nov 03 22:54:56.402077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.401944 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3fe12e9d-54ea-42b1-ba23-28edfd8826e7" containerName="revoke-keys" Nov 03 22:54:56.402077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.401951 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe12e9d-54ea-42b1-ba23-28edfd8826e7" containerName="revoke-keys" Nov 03 22:54:56.402340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.402095 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b072383-4e99-4952-a66a-1eeb8e0bffbc" containerName="revoke-keys" Nov 03 22:54:56.402340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.402110 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe12e9d-54ea-42b1-ba23-28edfd8826e7" containerName="revoke-keys" Nov 03 22:54:56.402340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.402133 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea3c3cb-4508-409d-afb4-a03249c5702b" containerName="payload-pre-processing" Nov 03 22:54:56.402340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.402142 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0c63e2-19da-49d8-84b7-f826385c2d68" containerName="payload-processing" Nov 03 22:54:56.407871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.407853 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.420784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.420714 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:54:56.421072 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.421049 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:54:56.421181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.421070 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:54:56.421181 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.421132 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:54:56.423584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.423561 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c"] Nov 03 22:54:56.496444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.496365 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7dbc2f7-n5j7c\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.496444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.496429 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lrc\" (UniqueName: \"kubernetes.io/projected/d6c57c73-7878-4cf9-88d3-45626c062b86-kube-api-access-q9lrc\") pod \"maas-api-revoke-sub-c7dbc2f7-n5j7c\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.496673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.496533 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7dbc2f7-n5j7c\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.606359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.600985 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7dbc2f7-n5j7c\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.606359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.601084 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lrc\" (UniqueName: \"kubernetes.io/projected/d6c57c73-7878-4cf9-88d3-45626c062b86-kube-api-access-q9lrc\") pod \"maas-api-revoke-sub-c7dbc2f7-n5j7c\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.606359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.601138 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7dbc2f7-n5j7c\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.606359 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.602085 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7dbc2f7-n5j7c\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.610689 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.610035 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7dbc2f7-n5j7c\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.626458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.626426 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lrc\" (UniqueName: \"kubernetes.io/projected/d6c57c73-7878-4cf9-88d3-45626c062b86-kube-api-access-q9lrc\") pod \"maas-api-revoke-sub-c7dbc2f7-n5j7c\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:56.727215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:56.727185 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:57.057301 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.057265 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c"] Nov 03 22:54:57.058696 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:54:57.058659 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c57c73_7878_4cf9_88d3_45626c062b86.slice/crio-3d4af7d00764f1110cb1eddd535131b3f6b759d6fe05c2df3a87d2055ba420c7 WatchSource:0}: Error finding container 3d4af7d00764f1110cb1eddd535131b3f6b759d6fe05c2df3a87d2055ba420c7: Status 404 returned error can't find the container with id 3d4af7d00764f1110cb1eddd535131b3f6b759d6fe05c2df3a87d2055ba420c7 Nov 03 22:54:57.244696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.244660 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" event={"ID":"d6c57c73-7878-4cf9-88d3-45626c062b86","Type":"ContainerStarted","Data":"3d4af7d00764f1110cb1eddd535131b3f6b759d6fe05c2df3a87d2055ba420c7"} Nov 03 22:54:57.531349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.531317 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg"] Nov 03 22:54:57.536473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.536447 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.554034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.553382 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-2a2eca-openshift-default-dockercfg-t2dd6\"" Nov 03 22:54:57.570066 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.570028 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg"] Nov 03 22:54:57.591393 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.591359 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8"] Nov 03 22:54:57.591681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.591624 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" containerID="cri-o://9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4" gracePeriod=30 Nov 03 22:54:57.614743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.614695 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-podinfo\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.614918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.614797 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6swdr\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-kube-api-access-6swdr\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.614918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.614863 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-data\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.614918 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.614894 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.615176 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.615146 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istiod-ca-cert\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.615244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.615194 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-certs\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.615356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.615280 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-token\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.615427 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.615374 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-credential-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.615488 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.615428 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-envoy\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.716573 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-envoy\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.716672 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-podinfo\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.716711 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6swdr\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-kube-api-access-6swdr\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.716754 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-data\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.716789 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.716990 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istiod-ca-cert\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.717018 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-certs\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.717054 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-token\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.717108 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-credential-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.717589 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-credential-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.718088 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-socket\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.719855 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istiod-ca-cert\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.720602 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-data\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.727366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.720868 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-certs\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.745321 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.739656 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-envoy\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.766319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.764212 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-podinfo\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.799017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.798980 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-token\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.801510 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.800272 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6swdr\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-kube-api-access-6swdr\") pod \"e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:57.801674 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.801640 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0c63e2-19da-49d8-84b7-f826385c2d68" path="/var/lib/kubelet/pods/7a0c63e2-19da-49d8-84b7-f826385c2d68/volumes" Nov 03 22:54:57.802341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.802287 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea3c3cb-4508-409d-afb4-a03249c5702b" path="/var/lib/kubelet/pods/cea3c3cb-4508-409d-afb4-a03249c5702b/volumes" Nov 03 22:54:57.871278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:57.871246 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:58.072001 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.071967 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v"] Nov 03 22:54:58.103724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.103696 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9"] Nov 03 22:54:58.195511 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.195480 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg"] Nov 03 22:54:58.196990 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:54:58.196961 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a9ee84_e898_4b08_8889_7f7c6c566ac0.slice/crio-a3d5278a422cce96b805f510c1ae3bc185c12e5d62e083607b55d3e1bf1e79e3 WatchSource:0}: Error finding container a3d5278a422cce96b805f510c1ae3bc185c12e5d62e083607b55d3e1bf1e79e3: Status 404 returned error can't find the container with id a3d5278a422cce96b805f510c1ae3bc185c12e5d62e083607b55d3e1bf1e79e3 Nov 03 22:54:58.199866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.199819 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:54:58.199987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.199905 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:54:58.199987 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.199949 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:54:58.273073 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.273043 4903 generic.go:356] "Generic (PLEG): container finished" podID="d6c57c73-7878-4cf9-88d3-45626c062b86" containerID="483e9c56ac371efdd86f32c7e41b21124c056e1d13dcc747cffc9c02c86218bd" exitCode=0 Nov 03 22:54:58.273227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.273125 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" event={"ID":"d6c57c73-7878-4cf9-88d3-45626c062b86","Type":"ContainerDied","Data":"483e9c56ac371efdd86f32c7e41b21124c056e1d13dcc747cffc9c02c86218bd"} Nov 03 22:54:58.275439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.275411 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" event={"ID":"35a9ee84-e898-4b08-8889-7f7c6c566ac0","Type":"ContainerStarted","Data":"a3d5278a422cce96b805f510c1ae3bc185c12e5d62e083607b55d3e1bf1e79e3"} Nov 03 22:54:58.943121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.943087 4903 scope.go:122] "RemoveContainer" containerID="85db199ca35d0d658df4d84eba4482fa32a0c917b1b574facc5b3cfa207b3f4f" Nov 03 22:54:58.957439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.956349 4903 scope.go:122] "RemoveContainer" containerID="df8cf465b5b05eb75ab06ea9014cdb3690bdd3ff2139251441ff474733022473" Nov 03 22:54:58.966962 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:58.966937 4903 scope.go:122] "RemoveContainer" containerID="85c88b9ad10254270846a523b142b7b877a8b6020cb1afa87d103ad77b50dc00" Nov 03 22:54:59.287049 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.286952 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" event={"ID":"35a9ee84-e898-4b08-8889-7f7c6c566ac0","Type":"ContainerStarted","Data":"57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa"} Nov 03 22:54:59.329688 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.329605 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" podStartSLOduration=2.329590173 podStartE2EDuration="2.329590173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:54:59.327838645 +0000 UTC m=+2525.731779620" watchObservedRunningTime="2026-11-03 22:54:59.329590173 +0000 UTC m=+2525.733531148" Nov 03 22:54:59.496553 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.496529 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:54:59.571077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.570973 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-openshift-ca\") pod \"d6c57c73-7878-4cf9-88d3-45626c062b86\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " Nov 03 22:54:59.571077 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.571051 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-xks-ca\") pod \"d6c57c73-7878-4cf9-88d3-45626c062b86\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " Nov 03 22:54:59.571376 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.571150 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c57c73-7878-4cf9-88d3-45626c062b86-kube-api-access-q9lrc\" (UniqueName: \"kubernetes.io/projected/d6c57c73-7878-4cf9-88d3-45626c062b86-kube-api-access-q9lrc\") pod \"d6c57c73-7878-4cf9-88d3-45626c062b86\" (UID: \"d6c57c73-7878-4cf9-88d3-45626c062b86\") " Nov 03 22:54:59.572344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.572314 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-openshift-ca" pod "d6c57c73-7878-4cf9-88d3-45626c062b86" (UID: "d6c57c73-7878-4cf9-88d3-45626c062b86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:54:59.575508 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.575480 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c57c73-7878-4cf9-88d3-45626c062b86-kube-api-access-q9lrc" pod "d6c57c73-7878-4cf9-88d3-45626c062b86" (UID: "d6c57c73-7878-4cf9-88d3-45626c062b86"). InnerVolumeSpecName "kube-api-access-q9lrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:54:59.578442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.578414 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-xks-ca" pod "d6c57c73-7878-4cf9-88d3-45626c062b86" (UID: "d6c57c73-7878-4cf9-88d3-45626c062b86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:54:59.673737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.673696 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:59.673737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.673733 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c57c73-7878-4cf9-88d3-45626c062b86-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:59.673958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.673748 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9lrc\" (UniqueName: \"kubernetes.io/projected/d6c57c73-7878-4cf9-88d3-45626c062b86-kube-api-access-q9lrc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:54:59.872521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.872437 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:54:59.874026 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.874003 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 22:54:59.874102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.874056 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 22:54:59.966343 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.965689 4903 patch_prober.go:29] interesting pod/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:54:59.966687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:54:59.966647 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:55:00.296797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.296765 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" Nov 03 22:55:00.297184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.296811 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c" event={"ID":"d6c57c73-7878-4cf9-88d3-45626c062b86","Type":"ContainerDied","Data":"3d4af7d00764f1110cb1eddd535131b3f6b759d6fe05c2df3a87d2055ba420c7"} Nov 03 22:55:00.297184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.296855 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d4af7d00764f1110cb1eddd535131b3f6b759d6fe05c2df3a87d2055ba420c7" Nov 03 22:55:00.866322 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.865110 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb"] Nov 03 22:55:00.872015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.871902 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d6c57c73-7878-4cf9-88d3-45626c062b86" containerName="revoke-keys" Nov 03 22:55:00.872015 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.871974 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c57c73-7878-4cf9-88d3-45626c062b86" containerName="revoke-keys" Nov 03 22:55:00.872255 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.872208 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c57c73-7878-4cf9-88d3-45626c062b86" containerName="revoke-keys" Nov 03 22:55:00.873150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.872750 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 22:55:00.873150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.872839 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 22:55:00.877440 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.877416 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb"] Nov 03 22:55:00.877593 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.877535 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:00.899474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:00.899451 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d8a3d1-openshift-default-dockercfg-c87pb\"" Nov 03 22:55:01.025791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.025756 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-envoy\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.025791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.025796 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istiod-ca-cert\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.026009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.025862 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-data\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.026009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.025904 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5ngr\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-kube-api-access-t5ngr\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.026009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.025983 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-certs\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.026114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.026037 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-podinfo\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.026114 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.026099 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.026195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.026130 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-token\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.026260 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.026239 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-credential-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.126964 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127022 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-token\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127050 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127052 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-credential-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127149 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-envoy\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127165 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istiod-ca-cert\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127344 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127255 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-data\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127340 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5ngr\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-kube-api-access-t5ngr\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127411 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127479 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-certs\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127539 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127525 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-podinfo\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127559 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-credential-socket\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127743 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127643 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-data\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127833 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-certs\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.127994 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.127976 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istiod-ca-cert\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.129750 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.129729 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-envoy\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.129933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.129916 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-podinfo\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.167953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.167910 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-token\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.169501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.169479 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5ngr\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-kube-api-access-t5ngr\") pod \"e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.195436 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.195389 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:01.568975 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:55:01.568941 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93642a17_3a9f_4321_ad02_4d9c26fa42c2.slice/crio-45985071e41afa41a74f1486f60eec03f48263fa8ae95bd49c3b9f7325be8fbd WatchSource:0}: Error finding container 45985071e41afa41a74f1486f60eec03f48263fa8ae95bd49c3b9f7325be8fbd: Status 404 returned error can't find the container with id 45985071e41afa41a74f1486f60eec03f48263fa8ae95bd49c3b9f7325be8fbd Nov 03 22:55:01.572138 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.572107 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:01.572220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.572173 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:01.572220 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.572204 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:01.573406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.573383 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb"] Nov 03 22:55:01.875574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.871723 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 22:55:01.875574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:01.871792 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 22:55:02.334007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:02.333966 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" event={"ID":"93642a17-3a9f-4321-ad02-4d9c26fa42c2","Type":"ContainerStarted","Data":"cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89"} Nov 03 22:55:02.334190 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:02.334033 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" event={"ID":"93642a17-3a9f-4321-ad02-4d9c26fa42c2","Type":"ContainerStarted","Data":"45985071e41afa41a74f1486f60eec03f48263fa8ae95bd49c3b9f7325be8fbd"} Nov 03 22:55:02.406600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:02.406528 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" podStartSLOduration=2.406508818 podStartE2EDuration="2.406508818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:02.402815588 +0000 UTC m=+2528.806756562" watchObservedRunningTime="2026-11-03 22:55:02.406508818 +0000 UTC m=+2528.810449798" Nov 03 22:55:02.876983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:02.876874 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 22:55:02.877746 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:02.877699 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 22:55:03.098468 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.098425 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" podUID="4b465382-7b48-44d4-9d7d-206d8aba6028" containerName="payload-processing" containerID="cri-o://5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06" gracePeriod=25 Nov 03 22:55:03.106215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.104080 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" podUID="4dfa9165-ca04-4b2a-a921-5422d1d8428f" containerName="payload-pre-processing" containerID="cri-o://df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7" gracePeriod=25 Nov 03 22:55:03.107623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.107602 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:55:03.131660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.131581 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-certs\") pod \"eb281e27-8506-4de9-91d3-4800a8f40dc3\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " Nov 03 22:55:03.131660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.131627 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-credential-socket\") pod \"eb281e27-8506-4de9-91d3-4800a8f40dc3\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " Nov 03 22:55:03.131896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.131667 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb281e27-8506-4de9-91d3-4800a8f40dc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb281e27-8506-4de9-91d3-4800a8f40dc3-istiod-ca-cert\") pod \"eb281e27-8506-4de9-91d3-4800a8f40dc3\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " Nov 03 22:55:03.131896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.131755 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-token\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-token\") pod \"eb281e27-8506-4de9-91d3-4800a8f40dc3\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " Nov 03 22:55:03.131896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.131799 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-kube-api-access-jzhwb\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-kube-api-access-jzhwb\") pod \"eb281e27-8506-4de9-91d3-4800a8f40dc3\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " Nov 03 22:55:03.131896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.131837 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-certs" pod "eb281e27-8506-4de9-91d3-4800a8f40dc3" (UID: "eb281e27-8506-4de9-91d3-4800a8f40dc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:03.132115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.131904 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-podinfo\") pod \"eb281e27-8506-4de9-91d3-4800a8f40dc3\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " Nov 03 22:55:03.132115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.131981 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-credential-socket" pod "eb281e27-8506-4de9-91d3-4800a8f40dc3" (UID: "eb281e27-8506-4de9-91d3-4800a8f40dc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:03.132115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.131995 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-envoy\") pod \"eb281e27-8506-4de9-91d3-4800a8f40dc3\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " Nov 03 22:55:03.132115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132047 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-socket\") pod \"eb281e27-8506-4de9-91d3-4800a8f40dc3\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " Nov 03 22:55:03.132115 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132084 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-data\") pod \"eb281e27-8506-4de9-91d3-4800a8f40dc3\" (UID: \"eb281e27-8506-4de9-91d3-4800a8f40dc3\") " Nov 03 22:55:03.132390 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132264 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-socket" pod "eb281e27-8506-4de9-91d3-4800a8f40dc3" (UID: "eb281e27-8506-4de9-91d3-4800a8f40dc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:03.132607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132492 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-data" pod "eb281e27-8506-4de9-91d3-4800a8f40dc3" (UID: "eb281e27-8506-4de9-91d3-4800a8f40dc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:03.132607 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132516 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb281e27-8506-4de9-91d3-4800a8f40dc3-istiod-ca-cert" pod "eb281e27-8506-4de9-91d3-4800a8f40dc3" (UID: "eb281e27-8506-4de9-91d3-4800a8f40dc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:03.132844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132751 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb281e27-8506-4de9-91d3-4800a8f40dc3-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:03.132844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132772 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:03.132844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132786 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:03.132844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132798 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:03.132844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.132810 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:03.134562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.134540 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-kube-api-access-jzhwb" pod "eb281e27-8506-4de9-91d3-4800a8f40dc3" (UID: "eb281e27-8506-4de9-91d3-4800a8f40dc3"). InnerVolumeSpecName "kube-api-access-jzhwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:03.134840 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.134822 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-podinfo" pod "eb281e27-8506-4de9-91d3-4800a8f40dc3" (UID: "eb281e27-8506-4de9-91d3-4800a8f40dc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:55:03.135007 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.134984 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-envoy" pod "eb281e27-8506-4de9-91d3-4800a8f40dc3" (UID: "eb281e27-8506-4de9-91d3-4800a8f40dc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:03.135083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.135007 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-token" pod "eb281e27-8506-4de9-91d3-4800a8f40dc3" (UID: "eb281e27-8506-4de9-91d3-4800a8f40dc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:03.201316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.198052 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:03.201316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.198213 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 22:55:03.201316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.198258 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 22:55:03.234398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.234353 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:03.234398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.234399 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhwb\" (UniqueName: \"kubernetes.io/projected/eb281e27-8506-4de9-91d3-4800a8f40dc3-kube-api-access-jzhwb\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:03.234639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.234414 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:03.234639 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.234428 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb281e27-8506-4de9-91d3-4800a8f40dc3-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:03.394127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.394028 4903 generic.go:356] "Generic (PLEG): container finished" podID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerID="9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4" exitCode=0 Nov 03 22:55:03.394285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.394234 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" Nov 03 22:55:03.394476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.394449 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" event={"ID":"eb281e27-8506-4de9-91d3-4800a8f40dc3","Type":"ContainerDied","Data":"9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4"} Nov 03 22:55:03.394620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.394512 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8" event={"ID":"eb281e27-8506-4de9-91d3-4800a8f40dc3","Type":"ContainerDied","Data":"56a23ee9253a861968e08e675d6decb6cc4ffde1b25a92fdfd6d134ed05896d4"} Nov 03 22:55:03.394620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.394546 4903 scope.go:122] "RemoveContainer" containerID="9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4" Nov 03 22:55:03.406943 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.406919 4903 scope.go:122] "RemoveContainer" containerID="afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b" Nov 03 22:55:03.418083 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.418061 4903 scope.go:122] "RemoveContainer" containerID="9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4" Nov 03 22:55:03.418462 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:55:03.418429 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4\": container with ID starting with 9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4 not found: ID does not exist" containerID="9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4" Nov 03 22:55:03.418542 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.418470 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4"} err="failed to get container status \"9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4\": rpc error: code = NotFound desc = could not find container \"9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4\": container with ID starting with 9b282ab352aecbed23e2d39c5e05ccc1d4375554d3c54cded13e778ae40ee5f4 not found: ID does not exist" Nov 03 22:55:03.418542 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.418489 4903 scope.go:122] "RemoveContainer" containerID="afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b" Nov 03 22:55:03.418817 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:55:03.418796 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b\": container with ID starting with afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b not found: ID does not exist" containerID="afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b" Nov 03 22:55:03.418878 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.418826 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b"} err="failed to get container status \"afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b\": rpc error: code = NotFound desc = could not find container \"afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b\": container with ID starting with afe14f846535f4213c8c5177960a0811745665f7effc7f718cb52b082099ab2b not found: ID does not exist" Nov 03 22:55:03.467319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.459933 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8"] Nov 03 22:55:03.496357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.495527 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-124490-openshift-default-54b4ffbf45-88rs8"] Nov 03 22:55:03.764317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.762464 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" path="/var/lib/kubelet/pods/eb281e27-8506-4de9-91d3-4800a8f40dc3/volumes" Nov 03 22:55:03.880320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.879412 4903 patch_prober.go:29] interesting pod/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 22:55:03.880320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:03.879478 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 22:55:04.196754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:04.196715 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 22:55:04.196945 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:04.196771 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 22:55:05.078843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:05.078642 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:55:05.079600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:05.079566 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:55:05.080640 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:05.080621 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 22:55:05.105516 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:05.105479 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" podUID="4b465382-7b48-44d4-9d7d-206d8aba6028" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:55:05.105681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:05.105517 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" podUID="4dfa9165-ca04-4b2a-a921-5422d1d8428f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:55:05.196861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:05.196822 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 22:55:05.197065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:05.196921 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 22:55:06.196603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:06.196566 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 22:55:06.196998 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:06.196623 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 22:55:07.195724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:07.195690 4903 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 22:55:07.195910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:07.195750 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 22:55:08.237445 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:08.237408 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:08.239833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:08.238845 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:08.239833 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:08.239502 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 22:55:15.106392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:15.106351 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" podUID="4b465382-7b48-44d4-9d7d-206d8aba6028" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:55:15.106819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:15.106392 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" podUID="4dfa9165-ca04-4b2a-a921-5422d1d8428f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 22:55:16.875779 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:16.875742 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6"] Nov 03 22:55:16.885257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:16.885226 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-963936-dc6f6"] Nov 03 22:55:17.738033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:17.737995 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe12e9d-54ea-42b1-ba23-28edfd8826e7" path="/var/lib/kubelet/pods/3fe12e9d-54ea-42b1-ba23-28edfd8826e7/volumes" Nov 03 22:55:18.385187 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:55:18.385148 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:55:20.683995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:20.683961 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd"] Nov 03 22:55:20.684606 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:20.684261 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" containerID="cri-o://ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c" gracePeriod=30 Nov 03 22:55:22.240090 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:22.240051 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2cwvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 22:55:22.240610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:22.240132 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 22:55:23.418507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.418482 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:55:23.434942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.434922 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:55:23.567426 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.567389 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b465382-7b48-44d4-9d7d-206d8aba6028-kube-api-access-kfkqk\" (UniqueName: \"kubernetes.io/projected/4b465382-7b48-44d4-9d7d-206d8aba6028-kube-api-access-kfkqk\") pod \"4b465382-7b48-44d4-9d7d-206d8aba6028\" (UID: \"4b465382-7b48-44d4-9d7d-206d8aba6028\") " Nov 03 22:55:23.567629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.567442 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfa9165-ca04-4b2a-a921-5422d1d8428f-kube-api-access-g58rg\" (UniqueName: \"kubernetes.io/projected/4dfa9165-ca04-4b2a-a921-5422d1d8428f-kube-api-access-g58rg\") pod \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\" (UID: \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\") " Nov 03 22:55:23.567629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.567564 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b465382-7b48-44d4-9d7d-206d8aba6028-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b465382-7b48-44d4-9d7d-206d8aba6028-plugins-config-volume\") pod \"4b465382-7b48-44d4-9d7d-206d8aba6028\" (UID: \"4b465382-7b48-44d4-9d7d-206d8aba6028\") " Nov 03 22:55:23.567629 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.567611 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfa9165-ca04-4b2a-a921-5422d1d8428f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfa9165-ca04-4b2a-a921-5422d1d8428f-plugins-config-volume\") pod \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\" (UID: \"4dfa9165-ca04-4b2a-a921-5422d1d8428f\") " Nov 03 22:55:23.568004 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.567980 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b465382-7b48-44d4-9d7d-206d8aba6028-plugins-config-volume" pod "4b465382-7b48-44d4-9d7d-206d8aba6028" (UID: "4b465382-7b48-44d4-9d7d-206d8aba6028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:23.568004 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.567992 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfa9165-ca04-4b2a-a921-5422d1d8428f-plugins-config-volume" pod "4dfa9165-ca04-4b2a-a921-5422d1d8428f" (UID: "4dfa9165-ca04-4b2a-a921-5422d1d8428f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:23.568225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.568198 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b465382-7b48-44d4-9d7d-206d8aba6028-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:23.568225 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.568227 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfa9165-ca04-4b2a-a921-5422d1d8428f-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:23.569720 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.569700 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b465382-7b48-44d4-9d7d-206d8aba6028-kube-api-access-kfkqk" pod "4b465382-7b48-44d4-9d7d-206d8aba6028" (UID: "4b465382-7b48-44d4-9d7d-206d8aba6028"). InnerVolumeSpecName "kube-api-access-kfkqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:23.569795 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.569730 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfa9165-ca04-4b2a-a921-5422d1d8428f-kube-api-access-g58rg" pod "4dfa9165-ca04-4b2a-a921-5422d1d8428f" (UID: "4dfa9165-ca04-4b2a-a921-5422d1d8428f"). InnerVolumeSpecName "kube-api-access-g58rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:23.651647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.651614 4903 generic.go:356] "Generic (PLEG): container finished" podID="4b465382-7b48-44d4-9d7d-206d8aba6028" containerID="5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06" exitCode=0 Nov 03 22:55:23.651849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.651661 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" Nov 03 22:55:23.651849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.651695 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" event={"ID":"4b465382-7b48-44d4-9d7d-206d8aba6028","Type":"ContainerDied","Data":"5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06"} Nov 03 22:55:23.651849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.651755 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v" event={"ID":"4b465382-7b48-44d4-9d7d-206d8aba6028","Type":"ContainerDied","Data":"1ad367fa8cd702f0f4e3161f075769d51d35170ed33e3549892203979ea595bb"} Nov 03 22:55:23.651849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.651784 4903 scope.go:122] "RemoveContainer" containerID="5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06" Nov 03 22:55:23.653319 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.653278 4903 generic.go:356] "Generic (PLEG): container finished" podID="4dfa9165-ca04-4b2a-a921-5422d1d8428f" containerID="df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7" exitCode=0 Nov 03 22:55:23.653433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.653319 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" event={"ID":"4dfa9165-ca04-4b2a-a921-5422d1d8428f","Type":"ContainerDied","Data":"df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7"} Nov 03 22:55:23.653433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.653348 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" Nov 03 22:55:23.653433 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.653372 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9" event={"ID":"4dfa9165-ca04-4b2a-a921-5422d1d8428f","Type":"ContainerDied","Data":"a99608722488be4a875ebefbe600bd391c87260ad10f0f102e28204b8bb17443"} Nov 03 22:55:23.664412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.664392 4903 scope.go:122] "RemoveContainer" containerID="5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06" Nov 03 22:55:23.664724 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:55:23.664702 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06\": container with ID starting with 5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06 not found: ID does not exist" containerID="5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06" Nov 03 22:55:23.664790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.664730 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06"} err="failed to get container status \"5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06\": rpc error: code = NotFound desc = could not find container \"5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06\": container with ID starting with 5ce34df2c5368a6cb082eaf4ada16e243a49d5bc29e58b7ae9389a2923b52b06 not found: ID does not exist" Nov 03 22:55:23.664790 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.664748 4903 scope.go:122] "RemoveContainer" containerID="df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7" Nov 03 22:55:23.669491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.669469 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfkqk\" (UniqueName: \"kubernetes.io/projected/4b465382-7b48-44d4-9d7d-206d8aba6028-kube-api-access-kfkqk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:23.669491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.669490 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g58rg\" (UniqueName: \"kubernetes.io/projected/4dfa9165-ca04-4b2a-a921-5422d1d8428f-kube-api-access-g58rg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:23.675194 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.675178 4903 scope.go:122] "RemoveContainer" containerID="df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7" Nov 03 22:55:23.675491 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:55:23.675472 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7\": container with ID starting with df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7 not found: ID does not exist" containerID="df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7" Nov 03 22:55:23.675552 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.675497 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7"} err="failed to get container status \"df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7\": rpc error: code = NotFound desc = could not find container \"df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7\": container with ID starting with df40b7307e412c130ac86a86f4f747dd13501e08f93989fae7e9b7e3d05ee9b7 not found: ID does not exist" Nov 03 22:55:23.701933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.701907 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v"] Nov 03 22:55:23.709569 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.709542 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-963936-589db57f54-5m29v"] Nov 03 22:55:23.736087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.736055 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b465382-7b48-44d4-9d7d-206d8aba6028" path="/var/lib/kubelet/pods/4b465382-7b48-44d4-9d7d-206d8aba6028/volumes" Nov 03 22:55:23.742514 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.742489 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9"] Nov 03 22:55:23.748252 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:23.748224 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-963936-6c6699ff5-mktm9"] Nov 03 22:55:25.735027 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:25.734995 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfa9165-ca04-4b2a-a921-5422d1d8428f" path="/var/lib/kubelet/pods/4dfa9165-ca04-4b2a-a921-5422d1d8428f/volumes" Nov 03 22:55:25.979028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:25.979000 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:55:26.103130 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103089 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-socket\") pod \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " Nov 03 22:55:26.103341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103139 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-token\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-token\") pod \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " Nov 03 22:55:26.103341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103186 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-credential-socket\") pod \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " Nov 03 22:55:26.103341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103245 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-data\") pod \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " Nov 03 22:55:26.103341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103266 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istiod-ca-cert\") pod \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " Nov 03 22:55:26.103341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103323 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-kube-api-access-9x8q4\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-kube-api-access-9x8q4\") pod \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " Nov 03 22:55:26.103678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103429 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-envoy\") pod \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " Nov 03 22:55:26.103678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103455 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-socket" pod "8c1a292c-ffde-4ae7-af34-d84fb47c6021" (UID: "8c1a292c-ffde-4ae7-af34-d84fb47c6021"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:26.103678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103491 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-podinfo\") pod \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " Nov 03 22:55:26.103678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103545 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-certs\") pod \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\" (UID: \"8c1a292c-ffde-4ae7-af34-d84fb47c6021\") " Nov 03 22:55:26.103678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103594 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-credential-socket" pod "8c1a292c-ffde-4ae7-af34-d84fb47c6021" (UID: "8c1a292c-ffde-4ae7-af34-d84fb47c6021"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:26.103952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103807 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-data" pod "8c1a292c-ffde-4ae7-af34-d84fb47c6021" (UID: "8c1a292c-ffde-4ae7-af34-d84fb47c6021"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:26.103952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.103878 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-certs" pod "8c1a292c-ffde-4ae7-af34-d84fb47c6021" (UID: "8c1a292c-ffde-4ae7-af34-d84fb47c6021"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:26.104071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.104042 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istiod-ca-cert" pod "8c1a292c-ffde-4ae7-af34-d84fb47c6021" (UID: "8c1a292c-ffde-4ae7-af34-d84fb47c6021"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:55:26.104233 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.104209 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:26.104341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.104236 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:26.104341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.104252 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:26.104341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.104266 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:26.104341 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.104279 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:26.105697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.105672 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-token" pod "8c1a292c-ffde-4ae7-af34-d84fb47c6021" (UID: "8c1a292c-ffde-4ae7-af34-d84fb47c6021"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:26.105824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.105759 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-kube-api-access-9x8q4" pod "8c1a292c-ffde-4ae7-af34-d84fb47c6021" (UID: "8c1a292c-ffde-4ae7-af34-d84fb47c6021"). InnerVolumeSpecName "kube-api-access-9x8q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:55:26.106017 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.106001 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-envoy" pod "8c1a292c-ffde-4ae7-af34-d84fb47c6021" (UID: "8c1a292c-ffde-4ae7-af34-d84fb47c6021"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:55:26.106065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.106035 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-podinfo" pod "8c1a292c-ffde-4ae7-af34-d84fb47c6021" (UID: "8c1a292c-ffde-4ae7-af34-d84fb47c6021"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 22:55:26.206013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.205977 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:26.206013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.206008 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:26.206013 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.206020 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:26.206281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.206029 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x8q4\" (UniqueName: \"kubernetes.io/projected/8c1a292c-ffde-4ae7-af34-d84fb47c6021-kube-api-access-9x8q4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:55:26.686024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.685992 4903 generic.go:356] "Generic (PLEG): container finished" podID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerID="ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c" exitCode=0 Nov 03 22:55:26.686215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.686042 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" Nov 03 22:55:26.686215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.686083 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" event={"ID":"8c1a292c-ffde-4ae7-af34-d84fb47c6021","Type":"ContainerDied","Data":"ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c"} Nov 03 22:55:26.686215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.686139 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd" event={"ID":"8c1a292c-ffde-4ae7-af34-d84fb47c6021","Type":"ContainerDied","Data":"db43d869040c9bdaa36e6638ca605e96d94cc9377464a1fcd8a677287c32a3ff"} Nov 03 22:55:26.686215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.686168 4903 scope.go:122] "RemoveContainer" containerID="ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c" Nov 03 22:55:26.698229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.698210 4903 scope.go:122] "RemoveContainer" containerID="ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c" Nov 03 22:55:26.698511 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:55:26.698492 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c\": container with ID starting with ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c not found: ID does not exist" containerID="ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c" Nov 03 22:55:26.698576 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.698519 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c"} err="failed to get container status \"ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c\": rpc error: code = NotFound desc = could not find container \"ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c\": container with ID starting with ab22b70d2f2bde20b5f38c2b17873353fd6cbc4279fdca07e874e880b7c0b96c not found: ID does not exist" Nov 03 22:55:26.736566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.736534 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd"] Nov 03 22:55:26.751271 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:26.751241 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2cwvd"] Nov 03 22:55:27.736272 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:27.736233 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" path="/var/lib/kubelet/pods/8c1a292c-ffde-4ae7-af34-d84fb47c6021/volumes" Nov 03 22:55:30.999938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:30.999901 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk"] Nov 03 22:55:31.000871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000849 4903 cpu_manager.go:414] "containerMap: removing container" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" Nov 03 22:55:31.000956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000873 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" Nov 03 22:55:31.000956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000902 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4dfa9165-ca04-4b2a-a921-5422d1d8428f" containerName="payload-pre-processing" Nov 03 22:55:31.000956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000909 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfa9165-ca04-4b2a-a921-5422d1d8428f" containerName="payload-pre-processing" Nov 03 22:55:31.000956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000936 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" Nov 03 22:55:31.000956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000945 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" Nov 03 22:55:31.001214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000963 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4b465382-7b48-44d4-9d7d-206d8aba6028" containerName="payload-processing" Nov 03 22:55:31.001214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000972 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b465382-7b48-44d4-9d7d-206d8aba6028" containerName="payload-processing" Nov 03 22:55:31.001214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000983 4903 cpu_manager.go:414] "containerMap: removing container" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" Nov 03 22:55:31.001214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.000990 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" Nov 03 22:55:31.001214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.001148 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfa9165-ca04-4b2a-a921-5422d1d8428f" containerName="payload-pre-processing" Nov 03 22:55:31.001214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.001165 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1a292c-ffde-4ae7-af34-d84fb47c6021" containerName="istio-proxy" Nov 03 22:55:31.001214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.001177 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b465382-7b48-44d4-9d7d-206d8aba6028" containerName="payload-processing" Nov 03 22:55:31.001214 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.001186 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" Nov 03 22:55:31.006716 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.006696 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:31.027569 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.027537 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w"] Nov 03 22:55:31.028734 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.028709 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb281e27-8506-4de9-91d3-4800a8f40dc3" containerName="istio-proxy" Nov 03 22:55:31.032935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.032913 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:31.035989 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.035961 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk"] Nov 03 22:55:31.036681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.036606 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-68410c-dockercfg-zp4tg\"" Nov 03 22:55:31.057331 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.052310 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-68410c\"" Nov 03 22:55:31.074349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.072567 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7adeadf-b12f-4560-8c90-2358c401fd5b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w\" (UID: \"b7adeadf-b12f-4560-8c90-2358c401fd5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:31.074349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.072726 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47r6t\" (UniqueName: \"kubernetes.io/projected/b6ba3694-698f-4cae-b1ff-6141160434ad-kube-api-access-47r6t\") pod \"payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk\" (UID: \"b6ba3694-698f-4cae-b1ff-6141160434ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:31.074349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.072861 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2nx\" (UniqueName: \"kubernetes.io/projected/b7adeadf-b12f-4560-8c90-2358c401fd5b-kube-api-access-dz2nx\") pod \"payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w\" (UID: \"b7adeadf-b12f-4560-8c90-2358c401fd5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:31.074349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.072905 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ba3694-698f-4cae-b1ff-6141160434ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk\" (UID: \"b6ba3694-698f-4cae-b1ff-6141160434ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:31.087976 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.087477 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w"] Nov 03 22:55:31.177317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.176441 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ba3694-698f-4cae-b1ff-6141160434ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk\" (UID: \"b6ba3694-698f-4cae-b1ff-6141160434ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:31.177317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.176621 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7adeadf-b12f-4560-8c90-2358c401fd5b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w\" (UID: \"b7adeadf-b12f-4560-8c90-2358c401fd5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:31.177317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.176691 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47r6t\" (UniqueName: \"kubernetes.io/projected/b6ba3694-698f-4cae-b1ff-6141160434ad-kube-api-access-47r6t\") pod \"payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk\" (UID: \"b6ba3694-698f-4cae-b1ff-6141160434ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:31.177317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.176821 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2nx\" (UniqueName: \"kubernetes.io/projected/b7adeadf-b12f-4560-8c90-2358c401fd5b-kube-api-access-dz2nx\") pod \"payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w\" (UID: \"b7adeadf-b12f-4560-8c90-2358c401fd5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:31.193817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.193230 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7adeadf-b12f-4560-8c90-2358c401fd5b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w\" (UID: \"b7adeadf-b12f-4560-8c90-2358c401fd5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:31.212009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.203601 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ba3694-698f-4cae-b1ff-6141160434ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk\" (UID: \"b6ba3694-698f-4cae-b1ff-6141160434ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:31.225638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.225605 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47r6t\" (UniqueName: \"kubernetes.io/projected/b6ba3694-698f-4cae-b1ff-6141160434ad-kube-api-access-47r6t\") pod \"payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk\" (UID: \"b6ba3694-698f-4cae-b1ff-6141160434ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:31.240358 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.239821 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2nx\" (UniqueName: \"kubernetes.io/projected/b7adeadf-b12f-4560-8c90-2358c401fd5b-kube-api-access-dz2nx\") pod \"payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w\" (UID: \"b7adeadf-b12f-4560-8c90-2358c401fd5b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:31.322502 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.322409 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:31.359006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.358966 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:31.690992 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:55:31.690949 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ba3694_698f_4cae_b1ff_6141160434ad.slice/crio-abc313820f0b2bc662b7034b28004e2264742ba5a7d3117ce74521e1812b3cac WatchSource:0}: Error finding container abc313820f0b2bc662b7034b28004e2264742ba5a7d3117ce74521e1812b3cac: Status 404 returned error can't find the container with id abc313820f0b2bc662b7034b28004e2264742ba5a7d3117ce74521e1812b3cac Nov 03 22:55:31.695351 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.695284 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk"] Nov 03 22:55:31.717091 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.717051 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w"] Nov 03 22:55:31.717834 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:55:31.717811 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7adeadf_b12f_4560_8c90_2358c401fd5b.slice/crio-55e92964ce0081c5728d5681c74f53122e453e5fcdf80ec9d6b5ed8403f7b80a WatchSource:0}: Error finding container 55e92964ce0081c5728d5681c74f53122e453e5fcdf80ec9d6b5ed8403f7b80a: Status 404 returned error can't find the container with id 55e92964ce0081c5728d5681c74f53122e453e5fcdf80ec9d6b5ed8403f7b80a Nov 03 22:55:31.794983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.794946 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" event={"ID":"b7adeadf-b12f-4560-8c90-2358c401fd5b","Type":"ContainerStarted","Data":"55e92964ce0081c5728d5681c74f53122e453e5fcdf80ec9d6b5ed8403f7b80a"} Nov 03 22:55:31.797454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:31.797422 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" event={"ID":"b6ba3694-698f-4cae-b1ff-6141160434ad","Type":"ContainerStarted","Data":"abc313820f0b2bc662b7034b28004e2264742ba5a7d3117ce74521e1812b3cac"} Nov 03 22:55:32.807238 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:32.807197 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" event={"ID":"b7adeadf-b12f-4560-8c90-2358c401fd5b","Type":"ContainerStarted","Data":"3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101"} Nov 03 22:55:32.807685 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:32.807278 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:32.809854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:32.809827 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" event={"ID":"b6ba3694-698f-4cae-b1ff-6141160434ad","Type":"ContainerStarted","Data":"5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547"} Nov 03 22:55:32.810014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:32.809893 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:32.847617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:32.847565 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" podStartSLOduration=2.84755081 podStartE2EDuration="2.84755081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:32.843377496 +0000 UTC m=+2559.247318471" watchObservedRunningTime="2026-11-03 22:55:32.84755081 +0000 UTC m=+2559.251491784" Nov 03 22:55:32.884065 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:32.884004 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" podStartSLOduration=2.883990862 podStartE2EDuration="2.883990862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:32.880166889 +0000 UTC m=+2559.284107865" watchObservedRunningTime="2026-11-03 22:55:32.883990862 +0000 UTC m=+2559.287931837" Nov 03 22:55:43.820775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:43.820692 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:55:43.820775 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:43.820772 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:55:48.384481 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:55:48.384438 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:55:53.386058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.386018 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4"] Nov 03 22:55:53.392127 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.392098 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.416656 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.416631 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4"] Nov 03 22:55:53.417817 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.417786 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3920cb-openshift-default-dockercfg-259xz\"" Nov 03 22:55:53.438540 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.438510 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zblt\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-kube-api-access-4zblt\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.438683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.438547 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-credential-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.438683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.438661 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-podinfo\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.438813 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.438792 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-envoy\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.438866 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.438849 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-token\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.438937 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.438920 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-certs\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.438979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.438964 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istiod-ca-cert\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.439035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.439018 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.439076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.439054 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-data\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.542137 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.542096 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-envoy\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.542422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.542401 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-token\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.542630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.542599 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-certs\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.542738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.542654 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istiod-ca-cert\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.542738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.542714 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.542851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.542743 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-data\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.542940 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.542903 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zblt\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-kube-api-access-4zblt\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.543020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.542958 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-credential-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.543079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.543029 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-podinfo\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.547094 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.547068 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istiod-ca-cert\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.559623 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.547589 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-data\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.559806 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.547851 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.559908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.547933 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-credential-socket\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.559908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.548133 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-certs\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.559908 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.558174 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-envoy\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.561088 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.561064 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-podinfo\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.583170 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.583136 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zblt\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-kube-api-access-4zblt\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.590169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.590136 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-token\") pod \"e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:53.712076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:53.712041 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:54.054235 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:54.051996 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4"] Nov 03 22:55:54.057075 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:54.057033 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:54.057188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:54.057115 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:54.057188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:54.057148 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:55:54.994708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:54.994669 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" event={"ID":"04577a03-c0d1-4798-93e7-edc5c3cf1a41","Type":"ContainerStarted","Data":"8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63"} Nov 03 22:55:54.995222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:54.994733 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" event={"ID":"04577a03-c0d1-4798-93e7-edc5c3cf1a41","Type":"ContainerStarted","Data":"080ef1f1414056c1f0b9bb8d4f94587874eb8c3d75c4eece4190d3f71f2b12aa"} Nov 03 22:55:55.048794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:55.048737 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" podStartSLOduration=2.04872274 podStartE2EDuration="2.04872274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:55:55.045380136 +0000 UTC m=+2581.449321113" watchObservedRunningTime="2026-11-03 22:55:55.04872274 +0000 UTC m=+2581.452663711" Nov 03 22:55:55.712240 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:55.712203 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:55.720012 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:55.719986 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:56.002378 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:56.002267 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:56.003384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:56.003363 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:55:59.086920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:55:59.086888 4903 scope.go:122] "RemoveContainer" containerID="1fba55516d01473198f5014ed853a9d4f0fa799a207c2a86ab2548b473236c02" Nov 03 22:56:01.034006 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:01.033975 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf"] Nov 03 22:56:01.043981 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:01.043950 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5767a5-psrqf"] Nov 03 22:56:01.047448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:01.047420 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm"] Nov 03 22:56:01.050471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:01.050447 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6c3b07-f9xvm"] Nov 03 22:56:01.736814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:01.736772 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13680af1-3551-4e3c-83fd-385621682d25" path="/var/lib/kubelet/pods/13680af1-3551-4e3c-83fd-385621682d25/volumes" Nov 03 22:56:01.737453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:01.737435 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789d7855-e65b-4f71-9ce6-f053a2500e98" path="/var/lib/kubelet/pods/789d7855-e65b-4f71-9ce6-f053a2500e98/volumes" Nov 03 22:56:15.033763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:15.033724 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt"] Nov 03 22:56:15.043022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:15.042995 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb188a10-8rpgt"] Nov 03 22:56:15.735952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:15.735913 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ed1b4e-3f61-407c-87b4-ad7e8ec65a20" path="/var/lib/kubelet/pods/53ed1b4e-3f61-407c-87b4-ad7e8ec65a20/volumes" Nov 03 22:56:18.384381 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:56:18.384346 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:56:20.028757 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:20.028714 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l"] Nov 03 22:56:20.038281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:20.038252 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53d0fa3f-b7q8l"] Nov 03 22:56:21.735281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:21.735243 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4144ae1-a247-4d89-be12-c10a665387ee" path="/var/lib/kubelet/pods/b4144ae1-a247-4d89-be12-c10a665387ee/volumes" Nov 03 22:56:25.771316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:25.752684 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:56:40.464051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:40.464015 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w"] Nov 03 22:56:40.491020 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:40.490988 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk"] Nov 03 22:56:45.465246 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.465198 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" podUID="b7adeadf-b12f-4560-8c90-2358c401fd5b" containerName="payload-processing" containerID="cri-o://3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101" gracePeriod=25 Nov 03 22:56:45.491732 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.491664 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" podUID="b6ba3694-698f-4cae-b1ff-6141160434ad" containerName="payload-pre-processing" containerID="cri-o://5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547" gracePeriod=25 Nov 03 22:56:45.800875 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.800847 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:56:45.841033 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.841005 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:56:45.995595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.995515 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7adeadf-b12f-4560-8c90-2358c401fd5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7adeadf-b12f-4560-8c90-2358c401fd5b-plugins-config-volume\") pod \"b7adeadf-b12f-4560-8c90-2358c401fd5b\" (UID: \"b7adeadf-b12f-4560-8c90-2358c401fd5b\") " Nov 03 22:56:45.995595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.995572 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7adeadf-b12f-4560-8c90-2358c401fd5b-kube-api-access-dz2nx\" (UniqueName: \"kubernetes.io/projected/b7adeadf-b12f-4560-8c90-2358c401fd5b-kube-api-access-dz2nx\") pod \"b7adeadf-b12f-4560-8c90-2358c401fd5b\" (UID: \"b7adeadf-b12f-4560-8c90-2358c401fd5b\") " Nov 03 22:56:45.995786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.995642 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ba3694-698f-4cae-b1ff-6141160434ad-kube-api-access-47r6t\" (UniqueName: \"kubernetes.io/projected/b6ba3694-698f-4cae-b1ff-6141160434ad-kube-api-access-47r6t\") pod \"b6ba3694-698f-4cae-b1ff-6141160434ad\" (UID: \"b6ba3694-698f-4cae-b1ff-6141160434ad\") " Nov 03 22:56:45.995786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.995704 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ba3694-698f-4cae-b1ff-6141160434ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ba3694-698f-4cae-b1ff-6141160434ad-plugins-config-volume\") pod \"b6ba3694-698f-4cae-b1ff-6141160434ad\" (UID: \"b6ba3694-698f-4cae-b1ff-6141160434ad\") " Nov 03 22:56:45.995923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.995900 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7adeadf-b12f-4560-8c90-2358c401fd5b-plugins-config-volume" pod "b7adeadf-b12f-4560-8c90-2358c401fd5b" (UID: "b7adeadf-b12f-4560-8c90-2358c401fd5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:45.996163 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.996136 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7adeadf-b12f-4560-8c90-2358c401fd5b-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:56:45.996244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.996167 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ba3694-698f-4cae-b1ff-6141160434ad-plugins-config-volume" pod "b6ba3694-698f-4cae-b1ff-6141160434ad" (UID: "b6ba3694-698f-4cae-b1ff-6141160434ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:56:45.997811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.997791 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7adeadf-b12f-4560-8c90-2358c401fd5b-kube-api-access-dz2nx" pod "b7adeadf-b12f-4560-8c90-2358c401fd5b" (UID: "b7adeadf-b12f-4560-8c90-2358c401fd5b"). InnerVolumeSpecName "kube-api-access-dz2nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:45.997881 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:45.997823 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ba3694-698f-4cae-b1ff-6141160434ad-kube-api-access-47r6t" pod "b6ba3694-698f-4cae-b1ff-6141160434ad" (UID: "b6ba3694-698f-4cae-b1ff-6141160434ad"). InnerVolumeSpecName "kube-api-access-47r6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:56:46.097236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.097195 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ba3694-698f-4cae-b1ff-6141160434ad-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:56:46.097236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.097228 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2nx\" (UniqueName: \"kubernetes.io/projected/b7adeadf-b12f-4560-8c90-2358c401fd5b-kube-api-access-dz2nx\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:56:46.097236 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.097244 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47r6t\" (UniqueName: \"kubernetes.io/projected/b6ba3694-698f-4cae-b1ff-6141160434ad-kube-api-access-47r6t\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:56:46.403722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.403693 4903 generic.go:356] "Generic (PLEG): container finished" podID="b6ba3694-698f-4cae-b1ff-6141160434ad" containerID="5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547" exitCode=0 Nov 03 22:56:46.403901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.403741 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" Nov 03 22:56:46.403901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.403779 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" event={"ID":"b6ba3694-698f-4cae-b1ff-6141160434ad","Type":"ContainerDied","Data":"5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547"} Nov 03 22:56:46.403901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.403836 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk" event={"ID":"b6ba3694-698f-4cae-b1ff-6141160434ad","Type":"ContainerDied","Data":"abc313820f0b2bc662b7034b28004e2264742ba5a7d3117ce74521e1812b3cac"} Nov 03 22:56:46.403901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.403863 4903 scope.go:122] "RemoveContainer" containerID="5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547" Nov 03 22:56:46.405469 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.405445 4903 generic.go:356] "Generic (PLEG): container finished" podID="b7adeadf-b12f-4560-8c90-2358c401fd5b" containerID="3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101" exitCode=0 Nov 03 22:56:46.405603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.405474 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" event={"ID":"b7adeadf-b12f-4560-8c90-2358c401fd5b","Type":"ContainerDied","Data":"3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101"} Nov 03 22:56:46.405603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.405500 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" Nov 03 22:56:46.405603 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.405526 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w" event={"ID":"b7adeadf-b12f-4560-8c90-2358c401fd5b","Type":"ContainerDied","Data":"55e92964ce0081c5728d5681c74f53122e453e5fcdf80ec9d6b5ed8403f7b80a"} Nov 03 22:56:46.417462 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.417443 4903 scope.go:122] "RemoveContainer" containerID="5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547" Nov 03 22:56:46.417754 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:56:46.417733 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547\": container with ID starting with 5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547 not found: ID does not exist" containerID="5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547" Nov 03 22:56:46.417813 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.417762 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547"} err="failed to get container status \"5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547\": rpc error: code = NotFound desc = could not find container \"5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547\": container with ID starting with 5f33972e30f4dba4ae1b40876f057ee719a6c10cce779fc3a00781b9036ea547 not found: ID does not exist" Nov 03 22:56:46.417813 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.417778 4903 scope.go:122] "RemoveContainer" containerID="3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101" Nov 03 22:56:46.428760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.428742 4903 scope.go:122] "RemoveContainer" containerID="3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101" Nov 03 22:56:46.429014 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:56:46.428987 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101\": container with ID starting with 3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101 not found: ID does not exist" containerID="3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101" Nov 03 22:56:46.429079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.429025 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101"} err="failed to get container status \"3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101\": rpc error: code = NotFound desc = could not find container \"3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101\": container with ID starting with 3dd935ab0d65bed6cd3d6d242104730c3a71deb893052f3436e7c2fc4135b101 not found: ID does not exist" Nov 03 22:56:46.451738 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.451709 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk"] Nov 03 22:56:46.461873 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.461845 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-76b7f58695-w8gpk"] Nov 03 22:56:46.487269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.487237 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w"] Nov 03 22:56:46.497748 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:46.497721 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-7f86bfb976-d9z7w"] Nov 03 22:56:47.736609 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:47.736569 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ba3694-698f-4cae-b1ff-6141160434ad" path="/var/lib/kubelet/pods/b6ba3694-698f-4cae-b1ff-6141160434ad/volumes" Nov 03 22:56:47.737118 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:47.737102 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7adeadf-b12f-4560-8c90-2358c401fd5b" path="/var/lib/kubelet/pods/b7adeadf-b12f-4560-8c90-2358c401fd5b/volumes" Nov 03 22:56:48.385158 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:56:48.385123 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:56:59.158386 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:59.158348 4903 scope.go:122] "RemoveContainer" containerID="137f7b4dde11509368607b233cd36530ccfb773ef27d817c3e51a0a83b4e1433" Nov 03 22:56:59.168215 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:56:59.168192 4903 scope.go:122] "RemoveContainer" containerID="f1bd3db269231439dfe49097cd91975a4e55426d22b4ae45aed1ca8550c2037a" Nov 03 22:57:12.113828 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.113745 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z"] Nov 03 22:57:12.114753 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.114734 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b7adeadf-b12f-4560-8c90-2358c401fd5b" containerName="payload-processing" Nov 03 22:57:12.114811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.114755 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7adeadf-b12f-4560-8c90-2358c401fd5b" containerName="payload-processing" Nov 03 22:57:12.114811 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.114803 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b6ba3694-698f-4cae-b1ff-6141160434ad" containerName="payload-pre-processing" Nov 03 22:57:12.114892 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.114811 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ba3694-698f-4cae-b1ff-6141160434ad" containerName="payload-pre-processing" Nov 03 22:57:12.114958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.114946 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ba3694-698f-4cae-b1ff-6141160434ad" containerName="payload-pre-processing" Nov 03 22:57:12.114994 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.114970 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7adeadf-b12f-4560-8c90-2358c401fd5b" containerName="payload-processing" Nov 03 22:57:12.120844 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.120820 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:12.136492 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.136462 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-68410c\"" Nov 03 22:57:12.136638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.136491 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-68410c-dockercfg-4rtbr\"" Nov 03 22:57:12.137584 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.137563 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z"] Nov 03 22:57:12.179009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.178973 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rwd\" (UniqueName: \"kubernetes.io/projected/05575897-8937-4040-97ea-3433a962f53b-kube-api-access-85rwd\") pod \"payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z\" (UID: \"05575897-8937-4040-97ea-3433a962f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:12.179253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.179229 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05575897-8937-4040-97ea-3433a962f53b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z\" (UID: \"05575897-8937-4040-97ea-3433a962f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:12.280835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.280785 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05575897-8937-4040-97ea-3433a962f53b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z\" (UID: \"05575897-8937-4040-97ea-3433a962f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:12.281023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.280935 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rwd\" (UniqueName: \"kubernetes.io/projected/05575897-8937-4040-97ea-3433a962f53b-kube-api-access-85rwd\") pod \"payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z\" (UID: \"05575897-8937-4040-97ea-3433a962f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:12.281507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.281489 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05575897-8937-4040-97ea-3433a962f53b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z\" (UID: \"05575897-8937-4040-97ea-3433a962f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:12.305314 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.305272 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr"] Nov 03 22:57:12.309891 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.309868 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:12.313955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.313929 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rwd\" (UniqueName: \"kubernetes.io/projected/05575897-8937-4040-97ea-3433a962f53b-kube-api-access-85rwd\") pod \"payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z\" (UID: \"05575897-8937-4040-97ea-3433a962f53b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:12.326643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.326615 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr"] Nov 03 22:57:12.382416 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.382281 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twcl7\" (UniqueName: \"kubernetes.io/projected/147459de-4415-4d95-b2a8-c3de6e4ecc8e-kube-api-access-twcl7\") pod \"payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr\" (UID: \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:12.382582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.382485 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147459de-4415-4d95-b2a8-c3de6e4ecc8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr\" (UID: \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:12.437040 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.437006 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:12.484108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.484067 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twcl7\" (UniqueName: \"kubernetes.io/projected/147459de-4415-4d95-b2a8-c3de6e4ecc8e-kube-api-access-twcl7\") pod \"payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr\" (UID: \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:12.484307 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.484167 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147459de-4415-4d95-b2a8-c3de6e4ecc8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr\" (UID: \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:12.484910 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.484892 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147459de-4415-4d95-b2a8-c3de6e4ecc8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr\" (UID: \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:12.507893 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.507862 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twcl7\" (UniqueName: \"kubernetes.io/projected/147459de-4415-4d95-b2a8-c3de6e4ecc8e-kube-api-access-twcl7\") pod \"payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr\" (UID: \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:12.639347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.639242 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:12.775467 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.775429 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z"] Nov 03 22:57:12.779073 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.779048 4903 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 22:57:12.970000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:12.969976 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr"] Nov 03 22:57:12.971670 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:57:12.971641 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147459de_4415_4d95_b2a8_c3de6e4ecc8e.slice/crio-712a5ae4b3e4d48a71e4552eae8808ded7b8ab9ff4ed3fc23cf40cc00e5fc6b5 WatchSource:0}: Error finding container 712a5ae4b3e4d48a71e4552eae8808ded7b8ab9ff4ed3fc23cf40cc00e5fc6b5: Status 404 returned error can't find the container with id 712a5ae4b3e4d48a71e4552eae8808ded7b8ab9ff4ed3fc23cf40cc00e5fc6b5 Nov 03 22:57:13.630971 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:13.630934 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" event={"ID":"05575897-8937-4040-97ea-3433a962f53b","Type":"ContainerStarted","Data":"3708ac75cac0ba0b7855991818f81e1b6b680eeb36271afca01501ea55716567"} Nov 03 22:57:13.633818 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:13.633781 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" event={"ID":"147459de-4415-4d95-b2a8-c3de6e4ecc8e","Type":"ContainerStarted","Data":"712a5ae4b3e4d48a71e4552eae8808ded7b8ab9ff4ed3fc23cf40cc00e5fc6b5"} Nov 03 22:57:15.656180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:15.656141 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" event={"ID":"05575897-8937-4040-97ea-3433a962f53b","Type":"ContainerStarted","Data":"1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80"} Nov 03 22:57:15.656672 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:15.656244 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:15.658152 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:15.658121 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" event={"ID":"147459de-4415-4d95-b2a8-c3de6e4ecc8e","Type":"ContainerStarted","Data":"d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43"} Nov 03 22:57:15.658320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:15.658211 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:15.698546 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:15.698487 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" podStartSLOduration=1.864368819 podStartE2EDuration="3.698472217s" totalImagesPullingTime="1.834103398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:15.694626683 +0000 UTC m=+2662.098567664" watchObservedRunningTime="2026-11-03 22:57:15.698472217 +0000 UTC m=+2662.102413192" Nov 03 22:57:15.734849 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:15.734775 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" podStartSLOduration=2.015188127 podStartE2EDuration="3.734756668s" totalImagesPullingTime="1.719568541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:15.733832132 +0000 UTC m=+2662.137773111" watchObservedRunningTime="2026-11-03 22:57:15.734756668 +0000 UTC m=+2662.138697644" Nov 03 22:57:17.699947 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:17.699908 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:57:18.390629 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:57:18.390593 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:57:21.920406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:21.920361 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd"] Nov 03 22:57:21.925204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:21.925182 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:21.953382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:21.953351 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd"] Nov 03 22:57:21.975745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:21.975708 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:57:21.975917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:21.975806 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:57:21.975991 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:21.975714 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:57:21.976101 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:21.976088 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:57:22.033025 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.032995 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6dnt\" (UniqueName: \"kubernetes.io/projected/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-kube-api-access-t6dnt\") pod \"maas-api-revoke-sub-11e687d8-f9zhd\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.033210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.033042 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e687d8-f9zhd\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.033210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.033103 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e687d8-f9zhd\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.134700 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.134667 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6dnt\" (UniqueName: \"kubernetes.io/projected/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-kube-api-access-t6dnt\") pod \"maas-api-revoke-sub-11e687d8-f9zhd\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.134903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.134723 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e687d8-f9zhd\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.134903 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.134768 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e687d8-f9zhd\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.135533 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.135510 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e687d8-f9zhd\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.137409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.137179 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e687d8-f9zhd\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.178810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.178742 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6dnt\" (UniqueName: \"kubernetes.io/projected/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-kube-api-access-t6dnt\") pod \"maas-api-revoke-sub-11e687d8-f9zhd\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.239678 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.239645 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:22.574340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.573026 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd"] Nov 03 22:57:22.574340 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:57:22.573602 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea8c72e_7ae7_4d64_a5db_fa6bb2e9a9b6.slice/crio-d674322fc6647f32f79770d71650dbcedefd33bed77b3bb501e16b9560f4a7e9 WatchSource:0}: Error finding container d674322fc6647f32f79770d71650dbcedefd33bed77b3bb501e16b9560f4a7e9: Status 404 returned error can't find the container with id d674322fc6647f32f79770d71650dbcedefd33bed77b3bb501e16b9560f4a7e9 Nov 03 22:57:22.735131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:22.735098 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" event={"ID":"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6","Type":"ContainerStarted","Data":"d674322fc6647f32f79770d71650dbcedefd33bed77b3bb501e16b9560f4a7e9"} Nov 03 22:57:23.770113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:23.770076 4903 generic.go:356] "Generic (PLEG): container finished" podID="fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6" containerID="4ffb30a12bc1fda202efcb58ac0011436f28585f0a278aab55ee9e87a286c9a8" exitCode=0 Nov 03 22:57:23.770796 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:23.770160 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" event={"ID":"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6","Type":"ContainerDied","Data":"4ffb30a12bc1fda202efcb58ac0011436f28585f0a278aab55ee9e87a286c9a8"} Nov 03 22:57:24.958024 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:24.957988 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:25.110448 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.110412 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-openshift-ca\") pod \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " Nov 03 22:57:25.110619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.110538 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-xks-ca\") pod \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " Nov 03 22:57:25.110619 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.110581 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-kube-api-access-t6dnt\" (UniqueName: \"kubernetes.io/projected/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-kube-api-access-t6dnt\") pod \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\" (UID: \"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6\") " Nov 03 22:57:25.110800 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.110774 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-openshift-ca" pod "fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6" (UID: "fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:25.110990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.110973 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:57:25.112858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.112826 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-xks-ca" pod "fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6" (UID: "fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:25.112968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.112931 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-kube-api-access-t6dnt" pod "fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6" (UID: "fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6"). InnerVolumeSpecName "kube-api-access-t6dnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:25.212188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.212154 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:57:25.212188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.212185 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6dnt\" (UniqueName: \"kubernetes.io/projected/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6-kube-api-access-t6dnt\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:57:25.794451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.794410 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" Nov 03 22:57:25.794451 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.794443 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd" event={"ID":"fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6","Type":"ContainerDied","Data":"d674322fc6647f32f79770d71650dbcedefd33bed77b3bb501e16b9560f4a7e9"} Nov 03 22:57:25.794696 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:25.794524 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d674322fc6647f32f79770d71650dbcedefd33bed77b3bb501e16b9560f4a7e9" Nov 03 22:57:26.667914 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:26.667884 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 22:57:26.668409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:26.668324 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 22:57:27.045491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:27.045393 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 22:57:27.045491 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:27.045444 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 22:57:29.827769 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:29.827735 4903 generic.go:356] "Generic (PLEG): container finished" podID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerID="8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63" exitCode=0 Nov 03 22:57:29.828227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:29.827803 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" event={"ID":"04577a03-c0d1-4798-93e7-edc5c3cf1a41","Type":"ContainerDied","Data":"8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63"} Nov 03 22:57:29.828349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:29.828225 4903 scope.go:122] "RemoveContainer" containerID="8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63" Nov 03 22:57:29.829956 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:29.829931 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:57:29.830042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:29.830001 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:57:29.830042 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:29.830034 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:57:30.029102 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:30.029069 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf"] Nov 03 22:57:30.032412 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:30.032383 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15ed03e7-95tlf"] Nov 03 22:57:30.837820 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:30.837785 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" event={"ID":"04577a03-c0d1-4798-93e7-edc5c3cf1a41","Type":"ContainerStarted","Data":"775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef"} Nov 03 22:57:31.712758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:31.712727 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:57:31.712964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:31.712861 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 03 22:57:31.712964 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:31.712934 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 03 22:57:31.736404 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:31.736375 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1ad657-16aa-48be-9d97-da10f06ab551" path="/var/lib/kubelet/pods/9c1ad657-16aa-48be-9d97-da10f06ab551/volumes" Nov 03 22:57:32.337620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.337580 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5"] Nov 03 22:57:32.338474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.338452 4903 cpu_manager.go:414] "containerMap: removing container" podUID="fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6" containerName="revoke-keys" Nov 03 22:57:32.338474 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.338474 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6" containerName="revoke-keys" Nov 03 22:57:32.338717 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.338703 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6" containerName="revoke-keys" Nov 03 22:57:32.368620 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.368589 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.369687 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.369658 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5"] Nov 03 22:57:32.381644 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.381619 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:57:32.381784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.381693 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:57:32.381993 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.381972 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:57:32.382071 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.382002 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:57:32.414353 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.414320 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qlf\" (UniqueName: \"kubernetes.io/projected/f67e4b26-0628-42c0-84cf-dd4d86637b49-kube-api-access-z7qlf\") pod \"maas-api-revoke-sub-5bd824a5-mdcl5\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.414501 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.414414 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bd824a5-mdcl5\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.414665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.414643 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bd824a5-mdcl5\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.516055 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.516024 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qlf\" (UniqueName: \"kubernetes.io/projected/f67e4b26-0628-42c0-84cf-dd4d86637b49-kube-api-access-z7qlf\") pod \"maas-api-revoke-sub-5bd824a5-mdcl5\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.516264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.516092 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bd824a5-mdcl5\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.516264 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.516228 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bd824a5-mdcl5\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.516838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.516811 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bd824a5-mdcl5\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.518538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.518514 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bd824a5-mdcl5\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.537731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.537703 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qlf\" (UniqueName: \"kubernetes.io/projected/f67e4b26-0628-42c0-84cf-dd4d86637b49-kube-api-access-z7qlf\") pod \"maas-api-revoke-sub-5bd824a5-mdcl5\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.684480 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.684437 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:32.713082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.713049 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 03 22:57:32.713248 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:32.713107 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 03 22:57:33.031731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:33.031700 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5"] Nov 03 22:57:33.032017 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:57:33.031997 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67e4b26_0628_42c0_84cf_dd4d86637b49.slice/crio-237ba15da70fbe8996a960e42d62fa06568cbb000406befeb1bc4d3ccd3bc73d WatchSource:0}: Error finding container 237ba15da70fbe8996a960e42d62fa06568cbb000406befeb1bc4d3ccd3bc73d: Status 404 returned error can't find the container with id 237ba15da70fbe8996a960e42d62fa06568cbb000406befeb1bc4d3ccd3bc73d Nov 03 22:57:33.713399 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:33.713367 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 03 22:57:33.713807 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:33.713418 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 03 22:57:33.891308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:33.891206 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" event={"ID":"f67e4b26-0628-42c0-84cf-dd4d86637b49","Type":"ContainerStarted","Data":"f8ea95357759c3b7f47b20e570c88520abf1405318cad2dacf13191a0c2904f1"} Nov 03 22:57:33.891308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:33.891267 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" event={"ID":"f67e4b26-0628-42c0-84cf-dd4d86637b49","Type":"ContainerStarted","Data":"237ba15da70fbe8996a960e42d62fa06568cbb000406befeb1bc4d3ccd3bc73d"} Nov 03 22:57:33.929992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:33.929943 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" podStartSLOduration=1.929928895 podStartE2EDuration="1.929928895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:33.927407954 +0000 UTC m=+2680.331348929" watchObservedRunningTime="2026-11-03 22:57:33.929928895 +0000 UTC m=+2680.333869870" Nov 03 22:57:34.713281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:34.713247 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 03 22:57:34.713500 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:34.713319 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 03 22:57:34.903373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:34.903339 4903 generic.go:356] "Generic (PLEG): container finished" podID="f67e4b26-0628-42c0-84cf-dd4d86637b49" containerID="f8ea95357759c3b7f47b20e570c88520abf1405318cad2dacf13191a0c2904f1" exitCode=0 Nov 03 22:57:34.903574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:34.903560 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" event={"ID":"f67e4b26-0628-42c0-84cf-dd4d86637b49","Type":"ContainerDied","Data":"f8ea95357759c3b7f47b20e570c88520abf1405318cad2dacf13191a0c2904f1"} Nov 03 22:57:34.904693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:34.904665 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc"] Nov 03 22:57:34.908281 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:34.908260 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:34.929797 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:34.929765 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ff80c0\"" Nov 03 22:57:34.929955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:34.929841 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ff80c0-dockercfg-x7pz5\"" Nov 03 22:57:34.936337 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:34.935585 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc"] Nov 03 22:57:35.135406 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.135325 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915578f7-944f-4a2c-a18d-15cc2a0ee34a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc\" (UID: \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:35.135651 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.135625 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95dkw\" (UniqueName: \"kubernetes.io/projected/915578f7-944f-4a2c-a18d-15cc2a0ee34a-kube-api-access-95dkw\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc\" (UID: \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:35.236885 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.236847 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95dkw\" (UniqueName: \"kubernetes.io/projected/915578f7-944f-4a2c-a18d-15cc2a0ee34a-kube-api-access-95dkw\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc\" (UID: \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:35.237118 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.236906 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915578f7-944f-4a2c-a18d-15cc2a0ee34a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc\" (UID: \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:35.237717 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.237693 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915578f7-944f-4a2c-a18d-15cc2a0ee34a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc\" (UID: \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:35.266346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.266279 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95dkw\" (UniqueName: \"kubernetes.io/projected/915578f7-944f-4a2c-a18d-15cc2a0ee34a-kube-api-access-95dkw\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc\" (UID: \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:35.521270 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.521238 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:35.712392 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.712355 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Nov 03 22:57:35.712573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.712422 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Nov 03 22:57:35.856107 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.856075 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc"] Nov 03 22:57:35.856563 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:57:35.856536 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915578f7_944f_4a2c_a18d_15cc2a0ee34a.slice/crio-7264b025fc0a3cd979a40a5fa8b38a625914ed071be975f990a0c772834e0f73 WatchSource:0}: Error finding container 7264b025fc0a3cd979a40a5fa8b38a625914ed071be975f990a0c772834e0f73: Status 404 returned error can't find the container with id 7264b025fc0a3cd979a40a5fa8b38a625914ed071be975f990a0c772834e0f73 Nov 03 22:57:35.913269 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:35.913236 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" event={"ID":"915578f7-944f-4a2c-a18d-15cc2a0ee34a","Type":"ContainerStarted","Data":"7264b025fc0a3cd979a40a5fa8b38a625914ed071be975f990a0c772834e0f73"} Nov 03 22:57:36.072458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.072432 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:36.170826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.170795 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67e4b26-0628-42c0-84cf-dd4d86637b49-kube-api-access-z7qlf\" (UniqueName: \"kubernetes.io/projected/f67e4b26-0628-42c0-84cf-dd4d86637b49-kube-api-access-z7qlf\") pod \"f67e4b26-0628-42c0-84cf-dd4d86637b49\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " Nov 03 22:57:36.171035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.170909 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-openshift-ca\") pod \"f67e4b26-0628-42c0-84cf-dd4d86637b49\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " Nov 03 22:57:36.171035 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.170965 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-xks-ca\") pod \"f67e4b26-0628-42c0-84cf-dd4d86637b49\" (UID: \"f67e4b26-0628-42c0-84cf-dd4d86637b49\") " Nov 03 22:57:36.171355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.171327 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-openshift-ca" pod "f67e4b26-0628-42c0-84cf-dd4d86637b49" (UID: "f67e4b26-0628-42c0-84cf-dd4d86637b49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:57:36.171535 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.171521 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:57:36.172977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.172956 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67e4b26-0628-42c0-84cf-dd4d86637b49-kube-api-access-z7qlf" pod "f67e4b26-0628-42c0-84cf-dd4d86637b49" (UID: "f67e4b26-0628-42c0-84cf-dd4d86637b49"). InnerVolumeSpecName "kube-api-access-z7qlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:57:36.173058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.173001 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-xks-ca" pod "f67e4b26-0628-42c0-84cf-dd4d86637b49" (UID: "f67e4b26-0628-42c0-84cf-dd4d86637b49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:57:36.273384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.273349 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f67e4b26-0628-42c0-84cf-dd4d86637b49-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:57:36.273384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.273379 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qlf\" (UniqueName: \"kubernetes.io/projected/f67e4b26-0628-42c0-84cf-dd4d86637b49-kube-api-access-z7qlf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:57:36.717366 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.717337 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:57:36.717843 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.717817 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:57:36.718574 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.718552 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 22:57:36.922965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.922935 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" Nov 03 22:57:36.922965 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.922951 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5" event={"ID":"f67e4b26-0628-42c0-84cf-dd4d86637b49","Type":"ContainerDied","Data":"237ba15da70fbe8996a960e42d62fa06568cbb000406befeb1bc4d3ccd3bc73d"} Nov 03 22:57:36.923473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.923007 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="237ba15da70fbe8996a960e42d62fa06568cbb000406befeb1bc4d3ccd3bc73d" Nov 03 22:57:36.925096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.925068 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" event={"ID":"915578f7-944f-4a2c-a18d-15cc2a0ee34a","Type":"ContainerStarted","Data":"c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d"} Nov 03 22:57:36.963553 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:36.963497 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" podStartSLOduration=2.963479671 podStartE2EDuration="2.963479671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:36.96111728 +0000 UTC m=+2683.365058256" watchObservedRunningTime="2026-11-03 22:57:36.963479671 +0000 UTC m=+2683.367420648" Nov 03 22:57:37.934159 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:37.934125 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:42.030622 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.030590 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p"] Nov 03 22:57:42.040229 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.040200 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2cae1c3-f9m5p"] Nov 03 22:57:42.587422 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.587392 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5"] Nov 03 22:57:42.588550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.588521 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f67e4b26-0628-42c0-84cf-dd4d86637b49" containerName="revoke-keys" Nov 03 22:57:42.588550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.588549 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67e4b26-0628-42c0-84cf-dd4d86637b49" containerName="revoke-keys" Nov 03 22:57:42.588739 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.588729 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67e4b26-0628-42c0-84cf-dd4d86637b49" containerName="revoke-keys" Nov 03 22:57:42.592053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.592016 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:42.608763 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.608705 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3920cb\"" Nov 03 22:57:42.608917 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.608875 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3920cb-dockercfg-dxt2l\"" Nov 03 22:57:42.614265 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.614241 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5"] Nov 03 22:57:42.669274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.669233 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwklp\" (UniqueName: \"kubernetes.io/projected/f210a637-436f-459e-9b41-fe49955a0e3c-kube-api-access-hwklp\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5\" (UID: \"f210a637-436f-459e-9b41-fe49955a0e3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:42.669507 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.669440 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f210a637-436f-459e-9b41-fe49955a0e3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5\" (UID: \"f210a637-436f-459e-9b41-fe49955a0e3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:42.773104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.773050 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwklp\" (UniqueName: \"kubernetes.io/projected/f210a637-436f-459e-9b41-fe49955a0e3c-kube-api-access-hwklp\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5\" (UID: \"f210a637-436f-459e-9b41-fe49955a0e3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:42.773287 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.773170 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f210a637-436f-459e-9b41-fe49955a0e3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5\" (UID: \"f210a637-436f-459e-9b41-fe49955a0e3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:42.773870 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.773846 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f210a637-436f-459e-9b41-fe49955a0e3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5\" (UID: \"f210a637-436f-459e-9b41-fe49955a0e3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:42.796803 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.796773 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2"] Nov 03 22:57:42.800377 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.800360 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:42.809869 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.809840 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwklp\" (UniqueName: \"kubernetes.io/projected/f210a637-436f-459e-9b41-fe49955a0e3c-kube-api-access-hwklp\") pod \"payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5\" (UID: \"f210a637-436f-459e-9b41-fe49955a0e3c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:42.841456 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.841387 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2"] Nov 03 22:57:42.874530 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.874495 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bd4dbd-7d51-4454-a43b-b8f5680c3857-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2\" (UID: \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:42.874723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.874632 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb96h\" (UniqueName: \"kubernetes.io/projected/53bd4dbd-7d51-4454-a43b-b8f5680c3857-kube-api-access-xb96h\") pod \"payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2\" (UID: \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:42.907871 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.907841 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:42.981705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.981668 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bd4dbd-7d51-4454-a43b-b8f5680c3857-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2\" (UID: \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:42.981935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.981771 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb96h\" (UniqueName: \"kubernetes.io/projected/53bd4dbd-7d51-4454-a43b-b8f5680c3857-kube-api-access-xb96h\") pod \"payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2\" (UID: \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:42.982923 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:42.982849 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bd4dbd-7d51-4454-a43b-b8f5680c3857-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2\" (UID: \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:43.005051 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.005025 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb96h\" (UniqueName: \"kubernetes.io/projected/53bd4dbd-7d51-4454-a43b-b8f5680c3857-kube-api-access-xb96h\") pod \"payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2\" (UID: \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:43.113819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.113733 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:43.212745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.212710 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5"] Nov 03 22:57:43.218214 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:57:43.218186 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf210a637_436f_459e_9b41_fe49955a0e3c.slice/crio-c28ae51b18b39cf86840de79ff1b2c2c399184c467c470373ad9126b01d82df7 WatchSource:0}: Error finding container c28ae51b18b39cf86840de79ff1b2c2c399184c467c470373ad9126b01d82df7: Status 404 returned error can't find the container with id c28ae51b18b39cf86840de79ff1b2c2c399184c467c470373ad9126b01d82df7 Nov 03 22:57:43.437608 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.437567 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2"] Nov 03 22:57:43.438706 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:57:43.438675 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bd4dbd_7d51_4454_a43b_b8f5680c3857.slice/crio-41951447c0714a15e23fe609d7335fdba0b3c4c118279ed2d20911fa203297dc WatchSource:0}: Error finding container 41951447c0714a15e23fe609d7335fdba0b3c4c118279ed2d20911fa203297dc: Status 404 returned error can't find the container with id 41951447c0714a15e23fe609d7335fdba0b3c4c118279ed2d20911fa203297dc Nov 03 22:57:43.736933 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.736852 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6bf448-b4ff-4082-a31f-c7a9062769e8" path="/var/lib/kubelet/pods/2d6bf448-b4ff-4082-a31f-c7a9062769e8/volumes" Nov 03 22:57:43.996195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.996097 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" event={"ID":"53bd4dbd-7d51-4454-a43b-b8f5680c3857","Type":"ContainerStarted","Data":"9b5eb9b6995b80769fab1f89cc42f8dcef1fcb13ba5d0077138d04023432d52d"} Nov 03 22:57:43.996195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.996160 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" event={"ID":"53bd4dbd-7d51-4454-a43b-b8f5680c3857","Type":"ContainerStarted","Data":"41951447c0714a15e23fe609d7335fdba0b3c4c118279ed2d20911fa203297dc"} Nov 03 22:57:43.996470 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.996212 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:43.998217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.998192 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" event={"ID":"f210a637-436f-459e-9b41-fe49955a0e3c","Type":"ContainerStarted","Data":"63438a1f51cfd71ae008bdc5bc25e03115edb624c8d47f632623492278957434"} Nov 03 22:57:43.998409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.998238 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" event={"ID":"f210a637-436f-459e-9b41-fe49955a0e3c","Type":"ContainerStarted","Data":"c28ae51b18b39cf86840de79ff1b2c2c399184c467c470373ad9126b01d82df7"} Nov 03 22:57:43.998409 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:43.998396 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:44.034596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:44.034497 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" podStartSLOduration=2.034480015 podStartE2EDuration="2.034480015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:44.03342665 +0000 UTC m=+2690.437367637" watchObservedRunningTime="2026-11-03 22:57:44.034480015 +0000 UTC m=+2690.438420991" Nov 03 22:57:44.073430 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:44.073384 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" podStartSLOduration=2.07337062 podStartE2EDuration="2.07337062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:57:44.070805109 +0000 UTC m=+2690.474746081" watchObservedRunningTime="2026-11-03 22:57:44.07337062 +0000 UTC m=+2690.477311591" Nov 03 22:57:48.386067 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:57:48.386030 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:57:48.946529 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:48.946495 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:57:54.323768 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:54.323733 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:57:54.327550 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:54.327525 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 22:57:54.331089 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:54.331068 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:57:54.334946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:54.334929 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 22:57:55.015439 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:55.015403 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 22:57:55.015632 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:55.015486 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 22:57:59.216336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:59.216275 4903 scope.go:122] "RemoveContainer" containerID="bd5b3375b1e05abbe6b85114e470c9194226ef444738be7850f58b1b814f5a59" Nov 03 22:57:59.227036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:59.227016 4903 scope.go:122] "RemoveContainer" containerID="1ff698031a37f99a5110f77529931915a5315640e7639be582beb1c7936c84f3" Nov 03 22:57:59.236847 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:59.236828 4903 scope.go:122] "RemoveContainer" containerID="810f64c76476af2dd0d608da191c538cb0733c9082d29b18268a5cf6bf730488" Nov 03 22:57:59.247108 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:59.247082 4903 scope.go:122] "RemoveContainer" containerID="9af89b580a3f3b860569541b0ee4ea9ee885ff36ce49894386c0e47693bc0f05" Nov 03 22:57:59.257842 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:57:59.257820 4903 scope.go:122] "RemoveContainer" containerID="1ab91ea7b14ca9a99bd18dca0e5fb92ebee60de11e7d167b8871a72ad7e27f2d" Nov 03 22:58:13.620263 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.620227 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld"] Nov 03 22:58:13.629167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.629134 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.655814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.655790 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-15e7a7-openshift-default-dockercfg-s6zns\"" Nov 03 22:58:13.660682 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.660643 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld"] Nov 03 22:58:13.784009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.783955 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83503d8f-0a17-4c64-84af-d93426654e30-istiod-ca-cert\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.784218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.784056 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83503d8f-0a17-4c64-84af-d93426654e30-istio-podinfo\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.784313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.784252 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-certs\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.784313 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.784276 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-envoy\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.784444 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.784363 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-credential-socket\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.784505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.784446 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfvnv\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-kube-api-access-bfvnv\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.784505 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.784474 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-istio-token\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.784628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.784562 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-data\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.784628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.784586 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-socket\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.886631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.886417 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-istio-token\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.886631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.886581 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-data\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.886631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.886614 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-socket\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.887000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.886640 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83503d8f-0a17-4c64-84af-d93426654e30-istiod-ca-cert\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.887000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.886706 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83503d8f-0a17-4c64-84af-d93426654e30-istio-podinfo\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.887000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.886886 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-certs\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.887000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.886940 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-envoy\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.887228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.887027 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-credential-socket\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.887228 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.887093 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfvnv\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-kube-api-access-bfvnv\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.888355 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.888051 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-data\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.888521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.888368 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-socket\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.889131 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.889101 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83503d8f-0a17-4c64-84af-d93426654e30-istiod-ca-cert\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.890721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.890694 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-credential-socket\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.890896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.890876 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-certs\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.892994 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.892971 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-envoy\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.895350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.894914 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83503d8f-0a17-4c64-84af-d93426654e30-istio-podinfo\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.908920 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.908898 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfvnv\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-kube-api-access-bfvnv\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.909029 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.908961 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-istio-token\") pod \"e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:13.956068 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:13.956037 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:14.279188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:14.279078 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld"] Nov 03 22:58:14.281701 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:58:14.281674 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83503d8f_0a17_4c64_84af_d93426654e30.slice/crio-7415ae54776c3091cfab100df09f33b69788c0cf35ade3146c7302b06da03b46 WatchSource:0}: Error finding container 7415ae54776c3091cfab100df09f33b69788c0cf35ade3146c7302b06da03b46: Status 404 returned error can't find the container with id 7415ae54776c3091cfab100df09f33b69788c0cf35ade3146c7302b06da03b46 Nov 03 22:58:14.291141 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:14.291099 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:58:14.291268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:14.291192 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:58:14.291268 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:14.291232 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:58:15.264379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:15.264344 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" event={"ID":"83503d8f-0a17-4c64-84af-d93426654e30","Type":"ContainerStarted","Data":"b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30"} Nov 03 22:58:15.264784 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:15.264404 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" event={"ID":"83503d8f-0a17-4c64-84af-d93426654e30","Type":"ContainerStarted","Data":"7415ae54776c3091cfab100df09f33b69788c0cf35ade3146c7302b06da03b46"} Nov 03 22:58:15.317924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:15.317862 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" podStartSLOduration=2.317849717 podStartE2EDuration="2.317849717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:15.315188792 +0000 UTC m=+2721.719129768" watchObservedRunningTime="2026-11-03 22:58:15.317849717 +0000 UTC m=+2721.721790692" Nov 03 22:58:15.957169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:15.957134 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:15.963195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:15.963169 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:16.271837 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:16.271755 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:16.273210 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:16.273191 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 22:58:18.385371 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:58:18.385336 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:58:23.028707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:23.028671 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r"] Nov 03 22:58:23.038802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:23.038771 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9f8b1-fld7r"] Nov 03 22:58:23.737038 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:23.737004 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7407c2-e51e-4bfb-aaa7-160ce693340d" path="/var/lib/kubelet/pods/bc7407c2-e51e-4bfb-aaa7-160ce693340d/volumes" Nov 03 22:58:33.626384 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:33.623536 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:58:41.374835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.374753 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4"] Nov 03 22:58:41.379776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.379757 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.393104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.393076 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:41.393253 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.393164 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:58:41.396278 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.396250 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4"] Nov 03 22:58:41.401952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.401924 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:58:41.402079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.402016 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:58:41.481618 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.481584 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmz7h\" (UniqueName: \"kubernetes.io/projected/2699ef44-af66-4fa3-8946-94eca4c5ff2e-kube-api-access-mmz7h\") pod \"maas-api-revoke-sub-fb87ff9e-qrqv4\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.481830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.481779 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb87ff9e-qrqv4\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.481830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.481822 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb87ff9e-qrqv4\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.582979 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.582939 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb87ff9e-qrqv4\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.583199 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.583121 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmz7h\" (UniqueName: \"kubernetes.io/projected/2699ef44-af66-4fa3-8946-94eca4c5ff2e-kube-api-access-mmz7h\") pod \"maas-api-revoke-sub-fb87ff9e-qrqv4\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.583347 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.583330 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb87ff9e-qrqv4\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.583672 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.583653 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb87ff9e-qrqv4\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.585730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.585711 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb87ff9e-qrqv4\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.604708 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.604681 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmz7h\" (UniqueName: \"kubernetes.io/projected/2699ef44-af66-4fa3-8946-94eca4c5ff2e-kube-api-access-mmz7h\") pod \"maas-api-revoke-sub-fb87ff9e-qrqv4\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:41.696067 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:41.696034 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:42.048876 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:58:42.048843 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2699ef44_af66_4fa3_8946_94eca4c5ff2e.slice/crio-7dfd61613f901efe4d02e88eb1ae28f4f02578e7a555a691b65a716a5d8c8eb1 WatchSource:0}: Error finding container 7dfd61613f901efe4d02e88eb1ae28f4f02578e7a555a691b65a716a5d8c8eb1: Status 404 returned error can't find the container with id 7dfd61613f901efe4d02e88eb1ae28f4f02578e7a555a691b65a716a5d8c8eb1 Nov 03 22:58:42.049034 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:42.048917 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4"] Nov 03 22:58:42.541532 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:42.541490 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" event={"ID":"2699ef44-af66-4fa3-8946-94eca4c5ff2e","Type":"ContainerStarted","Data":"a91098bbc44d946c23635a17dba9d50da9c0e3af6d6c8a7a6b453463f25d0808"} Nov 03 22:58:42.542036 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:42.541549 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" event={"ID":"2699ef44-af66-4fa3-8946-94eca4c5ff2e","Type":"ContainerStarted","Data":"7dfd61613f901efe4d02e88eb1ae28f4f02578e7a555a691b65a716a5d8c8eb1"} Nov 03 22:58:42.581327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:42.581255 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" podStartSLOduration=1.5812413520000002 podStartE2EDuration="1.581241352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:42.576995988 +0000 UTC m=+2748.980936963" watchObservedRunningTime="2026-11-03 22:58:42.581241352 +0000 UTC m=+2748.985182327" Nov 03 22:58:43.543520 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.543485 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6"] Nov 03 22:58:43.549398 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.549365 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.554595 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.554565 4903 generic.go:356] "Generic (PLEG): container finished" podID="2699ef44-af66-4fa3-8946-94eca4c5ff2e" containerID="a91098bbc44d946c23635a17dba9d50da9c0e3af6d6c8a7a6b453463f25d0808" exitCode=0 Nov 03 22:58:43.554756 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.554647 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" event={"ID":"2699ef44-af66-4fa3-8946-94eca4c5ff2e","Type":"ContainerDied","Data":"a91098bbc44d946c23635a17dba9d50da9c0e3af6d6c8a7a6b453463f25d0808"} Nov 03 22:58:43.571673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.571642 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6"] Nov 03 22:58:43.611142 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.611110 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-certs\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.611325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.611163 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.611325 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.611281 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-envoy\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.611428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.611385 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istiod-ca-cert\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.611428 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.611416 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-token\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.611579 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.611554 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-data\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.611657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.611635 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-credential-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.611750 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.611736 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-podinfo\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.611819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.611792 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26xxm\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-kube-api-access-26xxm\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.646611 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.646584 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ff80c0-openshift-default-dockercfg-v27gw\"" Nov 03 22:58:43.712952 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.712918 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-data\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.712958 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-credential-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713032 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-podinfo\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713121 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713090 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26xxm\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-kube-api-access-26xxm\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713168 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-certs\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713206 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713288 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713272 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-envoy\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713381 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istiod-ca-cert\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713421 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-token\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713631 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713570 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-data\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713648 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-credential-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713651 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-certs\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.713924 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.713903 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-socket\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.714195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.714177 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istiod-ca-cert\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.715973 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.715949 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-podinfo\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.716069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.715979 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-envoy\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.745442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.745411 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-token\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.745906 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.745887 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26xxm\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-kube-api-access-26xxm\") pod \"e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:43.868582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:43.868501 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:44.194896 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.194866 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6"] Nov 03 22:58:44.195425 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:58:44.195396 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f02f88d_4e82_4bf0_adea_351a83a49ca5.slice/crio-4ad882636a9ad374956bfcb4b564947ed3638df7d7897d00b983b047a3584650 WatchSource:0}: Error finding container 4ad882636a9ad374956bfcb4b564947ed3638df7d7897d00b983b047a3584650: Status 404 returned error can't find the container with id 4ad882636a9ad374956bfcb4b564947ed3638df7d7897d00b983b047a3584650 Nov 03 22:58:44.198583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.198551 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:58:44.198680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.198615 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:58:44.198680 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.198643 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 22:58:44.565554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.565466 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" event={"ID":"7f02f88d-4e82-4bf0-adea-351a83a49ca5","Type":"ContainerStarted","Data":"506f4628e7c5e6219e9ee67d7d33edf555cf576189cad7d87cf516047d60ade8"} Nov 03 22:58:44.565554 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.565524 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" event={"ID":"7f02f88d-4e82-4bf0-adea-351a83a49ca5","Type":"ContainerStarted","Data":"4ad882636a9ad374956bfcb4b564947ed3638df7d7897d00b983b047a3584650"} Nov 03 22:58:44.629001 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.628940 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" podStartSLOduration=1.628921401 podStartE2EDuration="1.628921401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:58:44.61689667 +0000 UTC m=+2751.020837638" watchObservedRunningTime="2026-11-03 22:58:44.628921401 +0000 UTC m=+2751.032862378" Nov 03 22:58:44.784315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.780673 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:44.828921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.828825 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2699ef44-af66-4fa3-8946-94eca4c5ff2e-kube-api-access-mmz7h\" (UniqueName: \"kubernetes.io/projected/2699ef44-af66-4fa3-8946-94eca4c5ff2e-kube-api-access-mmz7h\") pod \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " Nov 03 22:58:44.829087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.829009 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-openshift-ca\") pod \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " Nov 03 22:58:44.829087 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.829065 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-xks-ca\") pod \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\" (UID: \"2699ef44-af66-4fa3-8946-94eca4c5ff2e\") " Nov 03 22:58:44.830580 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.830554 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-openshift-ca" pod "2699ef44-af66-4fa3-8946-94eca4c5ff2e" (UID: "2699ef44-af66-4fa3-8946-94eca4c5ff2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:44.833053 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.832975 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-xks-ca" pod "2699ef44-af66-4fa3-8946-94eca4c5ff2e" (UID: "2699ef44-af66-4fa3-8946-94eca4c5ff2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:44.833267 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.833247 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2699ef44-af66-4fa3-8946-94eca4c5ff2e-kube-api-access-mmz7h" pod "2699ef44-af66-4fa3-8946-94eca4c5ff2e" (UID: "2699ef44-af66-4fa3-8946-94eca4c5ff2e"). InnerVolumeSpecName "kube-api-access-mmz7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:44.871349 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.870499 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:44.931614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.931579 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:58:44.931614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.931609 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2699ef44-af66-4fa3-8946-94eca4c5ff2e-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:58:44.931614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:44.931624 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmz7h\" (UniqueName: \"kubernetes.io/projected/2699ef44-af66-4fa3-8946-94eca4c5ff2e-kube-api-access-mmz7h\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:58:45.577483 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:45.577457 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" Nov 03 22:58:45.577902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:45.577498 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4" event={"ID":"2699ef44-af66-4fa3-8946-94eca4c5ff2e","Type":"ContainerDied","Data":"7dfd61613f901efe4d02e88eb1ae28f4f02578e7a555a691b65a716a5d8c8eb1"} Nov 03 22:58:45.577902 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:45.577552 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dfd61613f901efe4d02e88eb1ae28f4f02578e7a555a691b65a716a5d8c8eb1" Nov 03 22:58:45.871133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:45.871046 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 03 22:58:45.871133 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:45.871114 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 03 22:58:46.869241 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:46.869206 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 03 22:58:46.869728 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:46.869273 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 03 22:58:47.869165 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:47.869128 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 03 22:58:47.869475 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:47.869187 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 03 22:58:48.391707 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:58:48.391672 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:58:48.868880 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:48.868845 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 03 22:58:48.869069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:48.868908 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 03 22:58:49.869874 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:49.869835 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" start-of-body= Nov 03 22:58:49.870285 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:49.869925 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.44:15021/healthz/ready\": dial tcp 10.132.1.44:15021: connect: connection refused" Nov 03 22:58:50.874610 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:50.874578 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:50.875092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:50.875071 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:50.875861 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:50.875838 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 22:58:51.689350 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.689319 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4"] Nov 03 22:58:51.690076 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.690059 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2699ef44-af66-4fa3-8946-94eca4c5ff2e" containerName="revoke-keys" Nov 03 22:58:51.690145 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.690079 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2699ef44-af66-4fa3-8946-94eca4c5ff2e" containerName="revoke-keys" Nov 03 22:58:51.690217 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.690208 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2699ef44-af66-4fa3-8946-94eca4c5ff2e" containerName="revoke-keys" Nov 03 22:58:51.695110 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.695089 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:51.734096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.726557 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:58:51.734616 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.734508 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:58:51.734785 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.734640 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:58:51.741308 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.741261 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:58:51.780915 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.780884 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4"] Nov 03 22:58:51.787723 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.787685 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc7efbfe-84cz4\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:51.787884 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.787856 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc7efbfe-84cz4\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:51.788011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.787993 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlzvx\" (UniqueName: \"kubernetes.io/projected/a6f97c0b-8429-44fa-8926-fcb9ab390c20-kube-api-access-rlzvx\") pod \"maas-api-revoke-sub-fc7efbfe-84cz4\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:51.890005 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.889967 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlzvx\" (UniqueName: \"kubernetes.io/projected/a6f97c0b-8429-44fa-8926-fcb9ab390c20-kube-api-access-rlzvx\") pod \"maas-api-revoke-sub-fc7efbfe-84cz4\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:51.890472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.890218 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc7efbfe-84cz4\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:51.890472 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.890313 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc7efbfe-84cz4\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:51.890972 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.890947 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc7efbfe-84cz4\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:51.892648 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.892628 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc7efbfe-84cz4\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:51.911023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:51.911003 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlzvx\" (UniqueName: \"kubernetes.io/projected/a6f97c0b-8429-44fa-8926-fcb9ab390c20-kube-api-access-rlzvx\") pod \"maas-api-revoke-sub-fc7efbfe-84cz4\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:52.065824 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:52.065741 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:52.374570 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:52.374540 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4"] Nov 03 22:58:52.374897 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:58:52.374874 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f97c0b_8429_44fa_8926_fcb9ab390c20.slice/crio-a44636cf2f620c7a225d076eb742a92856db67cbf6af92bc5471526d03e46719 WatchSource:0}: Error finding container a44636cf2f620c7a225d076eb742a92856db67cbf6af92bc5471526d03e46719: Status 404 returned error can't find the container with id a44636cf2f620c7a225d076eb742a92856db67cbf6af92bc5471526d03e46719 Nov 03 22:58:52.657707 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:52.657671 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" event={"ID":"a6f97c0b-8429-44fa-8926-fcb9ab390c20","Type":"ContainerStarted","Data":"a44636cf2f620c7a225d076eb742a92856db67cbf6af92bc5471526d03e46719"} Nov 03 22:58:53.112340 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:53.112229 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc"] Nov 03 22:58:53.749654 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:53.749616 4903 generic.go:356] "Generic (PLEG): container finished" podID="a6f97c0b-8429-44fa-8926-fcb9ab390c20" containerID="ddda87662306b44f7c59eb058c1dc02f45c651e6db6102e876fcbc61ead07e8d" exitCode=0 Nov 03 22:58:53.768197 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:53.768160 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" event={"ID":"a6f97c0b-8429-44fa-8926-fcb9ab390c20","Type":"ContainerDied","Data":"ddda87662306b44f7c59eb058c1dc02f45c651e6db6102e876fcbc61ead07e8d"} Nov 03 22:58:54.943583 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:54.943536 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:54.958120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:54.958086 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f97c0b-8429-44fa-8926-fcb9ab390c20-kube-api-access-rlzvx\" (UniqueName: \"kubernetes.io/projected/a6f97c0b-8429-44fa-8926-fcb9ab390c20-kube-api-access-rlzvx\") pod \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " Nov 03 22:58:54.958330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:54.958180 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-openshift-ca\") pod \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " Nov 03 22:58:54.958330 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:54.958218 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-xks-ca\") pod \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\" (UID: \"a6f97c0b-8429-44fa-8926-fcb9ab390c20\") " Nov 03 22:58:54.958602 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:54.958574 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-openshift-ca" pod "a6f97c0b-8429-44fa-8926-fcb9ab390c20" (UID: "a6f97c0b-8429-44fa-8926-fcb9ab390c20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:54.958834 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:54.958817 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:58:54.960362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:54.960336 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-xks-ca" pod "a6f97c0b-8429-44fa-8926-fcb9ab390c20" (UID: "a6f97c0b-8429-44fa-8926-fcb9ab390c20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:58:54.960481 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:54.960470 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f97c0b-8429-44fa-8926-fcb9ab390c20-kube-api-access-rlzvx" pod "a6f97c0b-8429-44fa-8926-fcb9ab390c20" (UID: "a6f97c0b-8429-44fa-8926-fcb9ab390c20"). InnerVolumeSpecName "kube-api-access-rlzvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:55.059802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:55.059721 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlzvx\" (UniqueName: \"kubernetes.io/projected/a6f97c0b-8429-44fa-8926-fcb9ab390c20-kube-api-access-rlzvx\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:58:55.059802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:55.059752 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6f97c0b-8429-44fa-8926-fcb9ab390c20-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:58:55.769473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:55.769437 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" Nov 03 22:58:55.769473 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:55.769465 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4" event={"ID":"a6f97c0b-8429-44fa-8926-fcb9ab390c20","Type":"ContainerDied","Data":"a44636cf2f620c7a225d076eb742a92856db67cbf6af92bc5471526d03e46719"} Nov 03 22:58:55.769721 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:55.769512 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44636cf2f620c7a225d076eb742a92856db67cbf6af92bc5471526d03e46719" Nov 03 22:58:58.112681 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.112634 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" podUID="915578f7-944f-4a2c-a18d-15cc2a0ee34a" containerName="payload-pre-processing" containerID="cri-o://c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d" gracePeriod=25 Nov 03 22:58:58.403064 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.403036 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:58:58.506562 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.506533 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/915578f7-944f-4a2c-a18d-15cc2a0ee34a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915578f7-944f-4a2c-a18d-15cc2a0ee34a-plugins-config-volume\") pod \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\" (UID: \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\") " Nov 03 22:58:58.506733 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.506626 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915578f7-944f-4a2c-a18d-15cc2a0ee34a-kube-api-access-95dkw\" (UniqueName: \"kubernetes.io/projected/915578f7-944f-4a2c-a18d-15cc2a0ee34a-kube-api-access-95dkw\") pod \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\" (UID: \"915578f7-944f-4a2c-a18d-15cc2a0ee34a\") " Nov 03 22:58:58.507014 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.506980 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915578f7-944f-4a2c-a18d-15cc2a0ee34a-plugins-config-volume" pod "915578f7-944f-4a2c-a18d-15cc2a0ee34a" (UID: "915578f7-944f-4a2c-a18d-15cc2a0ee34a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:58:58.507167 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.507150 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915578f7-944f-4a2c-a18d-15cc2a0ee34a-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:58:58.508860 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.508830 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915578f7-944f-4a2c-a18d-15cc2a0ee34a-kube-api-access-95dkw" pod "915578f7-944f-4a2c-a18d-15cc2a0ee34a" (UID: "915578f7-944f-4a2c-a18d-15cc2a0ee34a"). InnerVolumeSpecName "kube-api-access-95dkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:58:58.608882 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.608848 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95dkw\" (UniqueName: \"kubernetes.io/projected/915578f7-944f-4a2c-a18d-15cc2a0ee34a-kube-api-access-95dkw\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:58:58.802084 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.802043 4903 generic.go:356] "Generic (PLEG): container finished" podID="915578f7-944f-4a2c-a18d-15cc2a0ee34a" containerID="c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d" exitCode=0 Nov 03 22:58:58.802401 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.802367 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" event={"ID":"915578f7-944f-4a2c-a18d-15cc2a0ee34a","Type":"ContainerDied","Data":"c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d"} Nov 03 22:58:58.802659 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.802638 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" event={"ID":"915578f7-944f-4a2c-a18d-15cc2a0ee34a","Type":"ContainerDied","Data":"7264b025fc0a3cd979a40a5fa8b38a625914ed071be975f990a0c772834e0f73"} Nov 03 22:58:58.802848 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.802832 4903 scope.go:122] "RemoveContainer" containerID="c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d" Nov 03 22:58:58.803169 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.802850 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc" Nov 03 22:58:58.821665 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.821643 4903 scope.go:122] "RemoveContainer" containerID="c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d" Nov 03 22:58:58.822012 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:58:58.821988 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d\": container with ID starting with c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d not found: ID does not exist" containerID="c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d" Nov 03 22:58:58.822104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.822023 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d"} err="failed to get container status \"c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d\": rpc error: code = NotFound desc = could not find container \"c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d\": container with ID starting with c18d622928741934634fa3137974a57bb37750fcd5a87255f3024cb2792a624d not found: ID does not exist" Nov 03 22:58:58.854835 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.854808 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc"] Nov 03 22:58:58.858258 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:58.858237 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5d8df5b89f-cp5nc"] Nov 03 22:58:59.309389 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:59.309357 4903 scope.go:122] "RemoveContainer" containerID="7ba2d07ea61bfe63a83237b086d9aa50fa167c16f1ff7dacff00d4624f848522" Nov 03 22:58:59.322731 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:59.322539 4903 scope.go:122] "RemoveContainer" containerID="4528b7404d663ceb0e3bf4055bfdd9b673ef9f909647a9af6f8e39446156c9b9" Nov 03 22:58:59.333069 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:59.333049 4903 scope.go:122] "RemoveContainer" containerID="32fac65ea85cd6b73e89f0ab66473a7f4411dee8bf83227fbc3411934b695680" Nov 03 22:58:59.757621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:58:59.757569 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915578f7-944f-4a2c-a18d-15cc2a0ee34a" path="/var/lib/kubelet/pods/915578f7-944f-4a2c-a18d-15cc2a0ee34a/volumes" Nov 03 22:59:01.030011 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:01.029975 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx"] Nov 03 22:59:01.039673 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:01.039642 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d589901a-d2vhx"] Nov 03 22:59:01.737548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:01.737506 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b778c9da-6e2b-4780-b4bc-895754557c01" path="/var/lib/kubelet/pods/b778c9da-6e2b-4780-b4bc-895754557c01/volumes" Nov 03 22:59:02.486737 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.486702 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh"] Nov 03 22:59:02.487464 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.487448 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a6f97c0b-8429-44fa-8926-fcb9ab390c20" containerName="revoke-keys" Nov 03 22:59:02.487525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.487464 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f97c0b-8429-44fa-8926-fcb9ab390c20" containerName="revoke-keys" Nov 03 22:59:02.487525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.487478 4903 cpu_manager.go:414] "containerMap: removing container" podUID="915578f7-944f-4a2c-a18d-15cc2a0ee34a" containerName="payload-pre-processing" Nov 03 22:59:02.487525 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.487483 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915578f7-944f-4a2c-a18d-15cc2a0ee34a" containerName="payload-pre-processing" Nov 03 22:59:02.487643 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.487633 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f97c0b-8429-44fa-8926-fcb9ab390c20" containerName="revoke-keys" Nov 03 22:59:02.487693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.487644 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="915578f7-944f-4a2c-a18d-15cc2a0ee34a" containerName="payload-pre-processing" Nov 03 22:59:02.493232 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.493211 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.510254 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.510229 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh"] Nov 03 22:59:02.515858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.515836 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:59:02.515858 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.515850 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:59:02.518560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.518053 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:59:02.518560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.518063 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:59:02.564120 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.564089 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd6d61d8-qf8sh\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.564315 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.564145 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stzp2\" (UniqueName: \"kubernetes.io/projected/e7164673-75cb-4497-b91e-8101d8abcbe8-kube-api-access-stzp2\") pod \"maas-api-revoke-sub-bd6d61d8-qf8sh\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.564369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.564287 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd6d61d8-qf8sh\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.665521 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.665464 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stzp2\" (UniqueName: \"kubernetes.io/projected/e7164673-75cb-4497-b91e-8101d8abcbe8-kube-api-access-stzp2\") pod \"maas-api-revoke-sub-bd6d61d8-qf8sh\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.665758 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.665599 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd6d61d8-qf8sh\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.665829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.665816 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd6d61d8-qf8sh\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.666259 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.666236 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd6d61d8-qf8sh\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.668098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.668079 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd6d61d8-qf8sh\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.687221 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.687189 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stzp2\" (UniqueName: \"kubernetes.io/projected/e7164673-75cb-4497-b91e-8101d8abcbe8-kube-api-access-stzp2\") pod \"maas-api-revoke-sub-bd6d61d8-qf8sh\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:02.807754 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:02.807673 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:03.119339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:03.114536 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh"] Nov 03 22:59:03.119339 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:59:03.115160 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7164673_75cb_4497_b91e_8101d8abcbe8.slice/crio-7c9ea029ec0b7440add3ff806d6cfeab3955fef127b9eed6d28ea106313e13c2 WatchSource:0}: Error finding container 7c9ea029ec0b7440add3ff806d6cfeab3955fef127b9eed6d28ea106313e13c2: Status 404 returned error can't find the container with id 7c9ea029ec0b7440add3ff806d6cfeab3955fef127b9eed6d28ea106313e13c2 Nov 03 22:59:03.841204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:03.841119 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" event={"ID":"e7164673-75cb-4497-b91e-8101d8abcbe8","Type":"ContainerStarted","Data":"00d10e790fe48223f1708792231c1b53f48c273079f2adec3988e0cb3f0ab1f3"} Nov 03 22:59:03.841204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:03.841182 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" event={"ID":"e7164673-75cb-4497-b91e-8101d8abcbe8","Type":"ContainerStarted","Data":"7c9ea029ec0b7440add3ff806d6cfeab3955fef127b9eed6d28ea106313e13c2"} Nov 03 22:59:03.878573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:03.878518 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" podStartSLOduration=1.87850062 podStartE2EDuration="1.87850062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:03.878183589 +0000 UTC m=+2770.282124565" watchObservedRunningTime="2026-11-03 22:59:03.87850062 +0000 UTC m=+2770.282441596" Nov 03 22:59:04.850453 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:04.850420 4903 generic.go:356] "Generic (PLEG): container finished" podID="e7164673-75cb-4497-b91e-8101d8abcbe8" containerID="00d10e790fe48223f1708792231c1b53f48c273079f2adec3988e0cb3f0ab1f3" exitCode=0 Nov 03 22:59:04.850857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:04.850518 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" event={"ID":"e7164673-75cb-4497-b91e-8101d8abcbe8","Type":"ContainerDied","Data":"00d10e790fe48223f1708792231c1b53f48c273079f2adec3988e0cb3f0ab1f3"} Nov 03 22:59:06.031495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.031469 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:06.115147 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.115057 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-openshift-ca\") pod \"e7164673-75cb-4497-b91e-8101d8abcbe8\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " Nov 03 22:59:06.115327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.115147 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-xks-ca\") pod \"e7164673-75cb-4497-b91e-8101d8abcbe8\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " Nov 03 22:59:06.115327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.115213 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7164673-75cb-4497-b91e-8101d8abcbe8-kube-api-access-stzp2\" (UniqueName: \"kubernetes.io/projected/e7164673-75cb-4497-b91e-8101d8abcbe8-kube-api-access-stzp2\") pod \"e7164673-75cb-4497-b91e-8101d8abcbe8\" (UID: \"e7164673-75cb-4497-b91e-8101d8abcbe8\") " Nov 03 22:59:06.115541 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.115511 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-openshift-ca" pod "e7164673-75cb-4497-b91e-8101d8abcbe8" (UID: "e7164673-75cb-4497-b91e-8101d8abcbe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:06.115853 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.115827 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:06.117429 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.117400 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7164673-75cb-4497-b91e-8101d8abcbe8-kube-api-access-stzp2" pod "e7164673-75cb-4497-b91e-8101d8abcbe8" (UID: "e7164673-75cb-4497-b91e-8101d8abcbe8"). InnerVolumeSpecName "kube-api-access-stzp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:06.122061 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.122023 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-xks-ca" pod "e7164673-75cb-4497-b91e-8101d8abcbe8" (UID: "e7164673-75cb-4497-b91e-8101d8abcbe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:06.217802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.217770 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7164673-75cb-4497-b91e-8101d8abcbe8-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:06.217802 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.217795 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stzp2\" (UniqueName: \"kubernetes.io/projected/e7164673-75cb-4497-b91e-8101d8abcbe8-kube-api-access-stzp2\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:06.867986 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.867953 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" Nov 03 22:59:06.868149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.867989 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh" event={"ID":"e7164673-75cb-4497-b91e-8101d8abcbe8","Type":"ContainerDied","Data":"7c9ea029ec0b7440add3ff806d6cfeab3955fef127b9eed6d28ea106313e13c2"} Nov 03 22:59:06.868149 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:06.868041 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c9ea029ec0b7440add3ff806d6cfeab3955fef127b9eed6d28ea106313e13c2" Nov 03 22:59:11.094838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.094800 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd"] Nov 03 22:59:11.095705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.095681 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e7164673-75cb-4497-b91e-8101d8abcbe8" containerName="revoke-keys" Nov 03 22:59:11.095705 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.095703 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7164673-75cb-4497-b91e-8101d8abcbe8" containerName="revoke-keys" Nov 03 22:59:11.095904 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.095892 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7164673-75cb-4497-b91e-8101d8abcbe8" containerName="revoke-keys" Nov 03 22:59:11.099931 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.099914 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.115886 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.115857 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd"] Nov 03 22:59:11.121538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.121506 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:59:11.121538 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.121514 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:59:11.121702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.121581 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:59:11.121702 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.121593 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:59:11.208548 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.208507 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmcd\" (UniqueName: \"kubernetes.io/projected/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-kube-api-access-chmcd\") pod \"maas-api-revoke-sub-acab1bd3-79pxd\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.208901 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.208872 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acab1bd3-79pxd\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.209016 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.208923 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acab1bd3-79pxd\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.310693 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.310656 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acab1bd3-79pxd\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.310846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.310727 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acab1bd3-79pxd\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.310916 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.310901 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmcd\" (UniqueName: \"kubernetes.io/projected/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-kube-api-access-chmcd\") pod \"maas-api-revoke-sub-acab1bd3-79pxd\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.311327 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.311286 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acab1bd3-79pxd\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.313180 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.313152 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acab1bd3-79pxd\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.332408 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.332379 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmcd\" (UniqueName: \"kubernetes.io/projected/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-kube-api-access-chmcd\") pod \"maas-api-revoke-sub-acab1bd3-79pxd\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.414995 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.414969 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:11.732716 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:59:11.732684 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeefbc1a_70c5_4a53_9440_2e6c07d5d6bb.slice/crio-8b9bd2939e8e11ee41ebfc602b8debabfe92d7e38a6db884a0b32f1ee8b5e53f WatchSource:0}: Error finding container 8b9bd2939e8e11ee41ebfc602b8debabfe92d7e38a6db884a0b32f1ee8b5e53f: Status 404 returned error can't find the container with id 8b9bd2939e8e11ee41ebfc602b8debabfe92d7e38a6db884a0b32f1ee8b5e53f Nov 03 22:59:11.737838 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.737811 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd"] Nov 03 22:59:11.924683 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:11.924648 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" event={"ID":"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb","Type":"ContainerStarted","Data":"8b9bd2939e8e11ee41ebfc602b8debabfe92d7e38a6db884a0b32f1ee8b5e53f"} Nov 03 22:59:12.934226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:12.934132 4903 generic.go:356] "Generic (PLEG): container finished" podID="deefbc1a-70c5-4a53-9440-2e6c07d5d6bb" containerID="3048e80298a1b5ce7e5ce93a192220f0ab8831658a622707fb287ca07186fd92" exitCode=0 Nov 03 22:59:12.934226 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:12.934196 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" event={"ID":"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb","Type":"ContainerDied","Data":"3048e80298a1b5ce7e5ce93a192220f0ab8831658a622707fb287ca07186fd92"} Nov 03 22:59:12.952184 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:12.952150 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw"] Nov 03 22:59:12.956668 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:12.956648 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:12.974275 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:12.974239 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw"] Nov 03 22:59:13.034759 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.034724 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141ec3d7-rfkvw\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.034946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.034899 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2t8x\" (UniqueName: \"kubernetes.io/projected/e5394e7f-9313-4a83-8721-42b8dbc40a34-kube-api-access-f2t8x\") pod \"maas-api-revoke-sub-141ec3d7-rfkvw\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.034946 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.034932 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141ec3d7-rfkvw\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.073638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.073606 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg"] Nov 03 22:59:13.078150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.078135 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.101374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.101348 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg"] Nov 03 22:59:13.136647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.136612 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141ec3d7-rfkvw\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.136857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.136726 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be86daa2-f9wxg\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.136857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.136756 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2t8x\" (UniqueName: \"kubernetes.io/projected/e5394e7f-9313-4a83-8721-42b8dbc40a34-kube-api-access-f2t8x\") pod \"maas-api-revoke-sub-141ec3d7-rfkvw\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.136857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.136775 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141ec3d7-rfkvw\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.137058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.136939 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxtf\" (UniqueName: \"kubernetes.io/projected/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-kube-api-access-5lxtf\") pod \"maas-api-revoke-sub-be86daa2-f9wxg\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.137058 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.136983 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be86daa2-f9wxg\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.137614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.137587 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141ec3d7-rfkvw\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.139338 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.139280 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141ec3d7-rfkvw\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.158735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.158694 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2t8x\" (UniqueName: \"kubernetes.io/projected/e5394e7f-9313-4a83-8721-42b8dbc40a34-kube-api-access-f2t8x\") pod \"maas-api-revoke-sub-141ec3d7-rfkvw\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.238745 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.238663 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be86daa2-f9wxg\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.238929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.238811 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxtf\" (UniqueName: \"kubernetes.io/projected/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-kube-api-access-5lxtf\") pod \"maas-api-revoke-sub-be86daa2-f9wxg\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.238929 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.238854 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be86daa2-f9wxg\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.239560 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.239528 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be86daa2-f9wxg\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.241360 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.241333 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be86daa2-f9wxg\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.264704 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.264676 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxtf\" (UniqueName: \"kubernetes.io/projected/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-kube-api-access-5lxtf\") pod \"maas-api-revoke-sub-be86daa2-f9wxg\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.271371 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.271349 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:13.392188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.392159 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:13.579980 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.579948 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw"] Nov 03 22:59:13.580982 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:59:13.580860 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5394e7f_9313_4a83_8721_42b8dbc40a34.slice/crio-dce2a3a28b54cb144c380ee46a328ce067fc739cfd48c461d19de5f7bb4fded4 WatchSource:0}: Error finding container dce2a3a28b54cb144c380ee46a328ce067fc739cfd48c461d19de5f7bb4fded4: Status 404 returned error can't find the container with id dce2a3a28b54cb144c380ee46a328ce067fc739cfd48c461d19de5f7bb4fded4 Nov 03 22:59:13.708876 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.708846 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg"] Nov 03 22:59:13.710505 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:59:13.710478 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d35cbb_4d75_494a_946d_1c5c7caf1b12.slice/crio-4e33b6fcf95cde9e472092ece67d85cd60bcec915777f0ccf3815a63d6bf9ec5 WatchSource:0}: Error finding container 4e33b6fcf95cde9e472092ece67d85cd60bcec915777f0ccf3815a63d6bf9ec5: Status 404 returned error can't find the container with id 4e33b6fcf95cde9e472092ece67d85cd60bcec915777f0ccf3815a63d6bf9ec5 Nov 03 22:59:13.944888 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.944851 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" event={"ID":"e5394e7f-9313-4a83-8721-42b8dbc40a34","Type":"ContainerStarted","Data":"7b9a3e393d37f90693ec0f94f53d0fa074db86cdf37033fe0fedbccdd8514fc6"} Nov 03 22:59:13.945353 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.944915 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" event={"ID":"e5394e7f-9313-4a83-8721-42b8dbc40a34","Type":"ContainerStarted","Data":"dce2a3a28b54cb144c380ee46a328ce067fc739cfd48c461d19de5f7bb4fded4"} Nov 03 22:59:13.946324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.946269 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" event={"ID":"c0d35cbb-4d75-494a-946d-1c5c7caf1b12","Type":"ContainerStarted","Data":"4e33b6fcf95cde9e472092ece67d85cd60bcec915777f0ccf3815a63d6bf9ec5"} Nov 03 22:59:13.985930 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:13.985873 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" podStartSLOduration=1.985857659 podStartE2EDuration="1.985857659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:13.981774729 +0000 UTC m=+2780.385715705" watchObservedRunningTime="2026-11-03 22:59:13.985857659 +0000 UTC m=+2780.389798633" Nov 03 22:59:14.096379 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.096354 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:14.151724 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.151698 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-kube-api-access-chmcd\" (UniqueName: \"kubernetes.io/projected/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-kube-api-access-chmcd\") pod \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " Nov 03 22:59:14.152156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.151870 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-openshift-ca\") pod \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " Nov 03 22:59:14.152156 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.151965 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-xks-ca\") pod \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\" (UID: \"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb\") " Nov 03 22:59:14.152776 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.152748 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-openshift-ca" pod "deefbc1a-70c5-4a53-9440-2e6c07d5d6bb" (UID: "deefbc1a-70c5-4a53-9440-2e6c07d5d6bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:14.154750 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.154717 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-xks-ca" pod "deefbc1a-70c5-4a53-9440-2e6c07d5d6bb" (UID: "deefbc1a-70c5-4a53-9440-2e6c07d5d6bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:14.155019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.154997 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-kube-api-access-chmcd" pod "deefbc1a-70c5-4a53-9440-2e6c07d5d6bb" (UID: "deefbc1a-70c5-4a53-9440-2e6c07d5d6bb"). InnerVolumeSpecName "kube-api-access-chmcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:14.253276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.253240 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:14.253276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.253273 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:14.253276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.253287 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chmcd\" (UniqueName: \"kubernetes.io/projected/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb-kube-api-access-chmcd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:14.956442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.956406 4903 generic.go:356] "Generic (PLEG): container finished" podID="e5394e7f-9313-4a83-8721-42b8dbc40a34" containerID="7b9a3e393d37f90693ec0f94f53d0fa074db86cdf37033fe0fedbccdd8514fc6" exitCode=0 Nov 03 22:59:14.956935 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.956498 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" event={"ID":"e5394e7f-9313-4a83-8721-42b8dbc40a34","Type":"ContainerDied","Data":"7b9a3e393d37f90693ec0f94f53d0fa074db86cdf37033fe0fedbccdd8514fc6"} Nov 03 22:59:14.958346 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.958317 4903 generic.go:356] "Generic (PLEG): container finished" podID="c0d35cbb-4d75-494a-946d-1c5c7caf1b12" containerID="fa56a9900bb90adba04d4bc32772016b8ff5ee2c330139569d3fd6c5dcf2d14b" exitCode=0 Nov 03 22:59:14.958513 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.958352 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" event={"ID":"c0d35cbb-4d75-494a-946d-1c5c7caf1b12","Type":"ContainerDied","Data":"fa56a9900bb90adba04d4bc32772016b8ff5ee2c330139569d3fd6c5dcf2d14b"} Nov 03 22:59:14.959839 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.959820 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" Nov 03 22:59:14.959942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.959836 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd" event={"ID":"deefbc1a-70c5-4a53-9440-2e6c07d5d6bb","Type":"ContainerDied","Data":"8b9bd2939e8e11ee41ebfc602b8debabfe92d7e38a6db884a0b32f1ee8b5e53f"} Nov 03 22:59:14.959942 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:14.959899 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b9bd2939e8e11ee41ebfc602b8debabfe92d7e38a6db884a0b32f1ee8b5e53f" Nov 03 22:59:16.144106 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.144073 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:16.179356 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.179317 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:16.181213 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.181080 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-kube-api-access-5lxtf\" (UniqueName: \"kubernetes.io/projected/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-kube-api-access-5lxtf\") pod \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " Nov 03 22:59:16.181415 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.181389 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-xks-ca\") pod \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " Nov 03 22:59:16.181547 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.181436 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-openshift-ca\") pod \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\" (UID: \"c0d35cbb-4d75-494a-946d-1c5c7caf1b12\") " Nov 03 22:59:16.183334 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.182932 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-openshift-ca" pod "c0d35cbb-4d75-494a-946d-1c5c7caf1b12" (UID: "c0d35cbb-4d75-494a-946d-1c5c7caf1b12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:16.187382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.187335 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-xks-ca" pod "c0d35cbb-4d75-494a-946d-1c5c7caf1b12" (UID: "c0d35cbb-4d75-494a-946d-1c5c7caf1b12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:16.187558 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.187417 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-kube-api-access-5lxtf" pod "c0d35cbb-4d75-494a-946d-1c5c7caf1b12" (UID: "c0d35cbb-4d75-494a-946d-1c5c7caf1b12"). InnerVolumeSpecName "kube-api-access-5lxtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:16.284227 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.284130 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-xks-ca\") pod \"e5394e7f-9313-4a83-8721-42b8dbc40a34\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " Nov 03 22:59:16.284459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.284395 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5394e7f-9313-4a83-8721-42b8dbc40a34-kube-api-access-f2t8x\" (UniqueName: \"kubernetes.io/projected/e5394e7f-9313-4a83-8721-42b8dbc40a34-kube-api-access-f2t8x\") pod \"e5394e7f-9313-4a83-8721-42b8dbc40a34\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " Nov 03 22:59:16.284459 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.284446 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-openshift-ca\") pod \"e5394e7f-9313-4a83-8721-42b8dbc40a34\" (UID: \"e5394e7f-9313-4a83-8721-42b8dbc40a34\") " Nov 03 22:59:16.285000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.284973 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:16.285000 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.284994 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:16.285195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.285009 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lxtf\" (UniqueName: \"kubernetes.io/projected/c0d35cbb-4d75-494a-946d-1c5c7caf1b12-kube-api-access-5lxtf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:16.285195 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.285064 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-openshift-ca" pod "e5394e7f-9313-4a83-8721-42b8dbc40a34" (UID: "e5394e7f-9313-4a83-8721-42b8dbc40a34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:16.287883 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.287857 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5394e7f-9313-4a83-8721-42b8dbc40a34-kube-api-access-f2t8x" pod "e5394e7f-9313-4a83-8721-42b8dbc40a34" (UID: "e5394e7f-9313-4a83-8721-42b8dbc40a34"). InnerVolumeSpecName "kube-api-access-f2t8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:16.288157 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.288135 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-xks-ca" pod "e5394e7f-9313-4a83-8721-42b8dbc40a34" (UID: "e5394e7f-9313-4a83-8721-42b8dbc40a34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:16.386092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.386056 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:16.386092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.386088 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2t8x\" (UniqueName: \"kubernetes.io/projected/e5394e7f-9313-4a83-8721-42b8dbc40a34-kube-api-access-f2t8x\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:16.386092 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.386099 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5394e7f-9313-4a83-8721-42b8dbc40a34-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:16.977218 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.977184 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" Nov 03 22:59:16.977471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.977184 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw" event={"ID":"e5394e7f-9313-4a83-8721-42b8dbc40a34","Type":"ContainerDied","Data":"dce2a3a28b54cb144c380ee46a328ce067fc739cfd48c461d19de5f7bb4fded4"} Nov 03 22:59:16.977471 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.977328 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dce2a3a28b54cb144c380ee46a328ce067fc739cfd48c461d19de5f7bb4fded4" Nov 03 22:59:16.978808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.978786 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" Nov 03 22:59:16.978808 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.978793 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg" event={"ID":"c0d35cbb-4d75-494a-946d-1c5c7caf1b12","Type":"ContainerDied","Data":"4e33b6fcf95cde9e472092ece67d85cd60bcec915777f0ccf3815a63d6bf9ec5"} Nov 03 22:59:16.978977 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:16.978842 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e33b6fcf95cde9e472092ece67d85cd60bcec915777f0ccf3815a63d6bf9ec5" Nov 03 22:59:18.386352 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:59:18.386280 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:59:21.502059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.502028 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26"] Nov 03 22:59:21.502992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.502970 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e5394e7f-9313-4a83-8721-42b8dbc40a34" containerName="revoke-keys" Nov 03 22:59:21.502992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.502993 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5394e7f-9313-4a83-8721-42b8dbc40a34" containerName="revoke-keys" Nov 03 22:59:21.503188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.503041 4903 cpu_manager.go:414] "containerMap: removing container" podUID="deefbc1a-70c5-4a53-9440-2e6c07d5d6bb" containerName="revoke-keys" Nov 03 22:59:21.503188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.503049 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deefbc1a-70c5-4a53-9440-2e6c07d5d6bb" containerName="revoke-keys" Nov 03 22:59:21.503188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.503082 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c0d35cbb-4d75-494a-946d-1c5c7caf1b12" containerName="revoke-keys" Nov 03 22:59:21.503188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.503089 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d35cbb-4d75-494a-946d-1c5c7caf1b12" containerName="revoke-keys" Nov 03 22:59:21.503457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.503255 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d35cbb-4d75-494a-946d-1c5c7caf1b12" containerName="revoke-keys" Nov 03 22:59:21.503457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.503276 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5394e7f-9313-4a83-8721-42b8dbc40a34" containerName="revoke-keys" Nov 03 22:59:21.503457 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.503312 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="deefbc1a-70c5-4a53-9440-2e6c07d5d6bb" containerName="revoke-keys" Nov 03 22:59:21.508537 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.508515 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.524600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.524574 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26"] Nov 03 22:59:21.528955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.528915 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:59:21.528955 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.528931 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:59:21.529126 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.528964 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:59:21.531690 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.531667 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:59:21.569177 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.569142 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzzn2\" (UniqueName: \"kubernetes.io/projected/003f2579-8b55-4491-abdc-83140f52caba-kube-api-access-jzzn2\") pod \"maas-api-revoke-sub-76b008d5-z9n26\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.569374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.569315 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f2579-8b55-4491-abdc-83140f52caba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76b008d5-z9n26\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.569374 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.569350 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f2579-8b55-4491-abdc-83140f52caba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76b008d5-z9n26\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.670819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.670788 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzzn2\" (UniqueName: \"kubernetes.io/projected/003f2579-8b55-4491-abdc-83140f52caba-kube-api-access-jzzn2\") pod \"maas-api-revoke-sub-76b008d5-z9n26\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.671023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.670938 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f2579-8b55-4491-abdc-83140f52caba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76b008d5-z9n26\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.671023 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.670971 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f2579-8b55-4491-abdc-83140f52caba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76b008d5-z9n26\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.671670 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.671650 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f2579-8b55-4491-abdc-83140f52caba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76b008d5-z9n26\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.673497 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.673479 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f2579-8b55-4491-abdc-83140f52caba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76b008d5-z9n26\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.692104 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.692076 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzzn2\" (UniqueName: \"kubernetes.io/projected/003f2579-8b55-4491-abdc-83140f52caba-kube-api-access-jzzn2\") pod \"maas-api-revoke-sub-76b008d5-z9n26\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:21.824591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:21.824511 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:22.129364 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:22.129338 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26"] Nov 03 22:59:22.129990 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:59:22.129964 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003f2579_8b55_4491_abdc_83140f52caba.slice/crio-497640d3864036563e609c7f3289f65398d0a16bf77cb9f0234e9513a12afee1 WatchSource:0}: Error finding container 497640d3864036563e609c7f3289f65398d0a16bf77cb9f0234e9513a12afee1: Status 404 returned error can't find the container with id 497640d3864036563e609c7f3289f65398d0a16bf77cb9f0234e9513a12afee1 Nov 03 22:59:23.028022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.027984 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" event={"ID":"003f2579-8b55-4491-abdc-83140f52caba","Type":"ContainerStarted","Data":"076e3665b8ed4fcfb8f3b75ee97d0dc14fe0726d05c3bf9d94217c44a3ae868a"} Nov 03 22:59:23.028449 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.028045 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" event={"ID":"003f2579-8b55-4491-abdc-83140f52caba","Type":"ContainerStarted","Data":"497640d3864036563e609c7f3289f65398d0a16bf77cb9f0234e9513a12afee1"} Nov 03 22:59:23.067605 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.067556 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" podStartSLOduration=2.067540293 podStartE2EDuration="2.067540293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:23.06414704 +0000 UTC m=+2789.468088019" watchObservedRunningTime="2026-11-03 22:59:23.067540293 +0000 UTC m=+2789.471481268" Nov 03 22:59:23.477188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.477137 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt"] Nov 03 22:59:23.482601 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.482579 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.491173 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.491144 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt"] Nov 03 22:59:23.597819 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.597783 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1207b689-tqskt\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.597996 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.597850 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1207b689-tqskt\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.598059 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.597990 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgqb9\" (UniqueName: \"kubernetes.io/projected/7004e68d-f98a-4945-bf5b-2d7261c5fb37-kube-api-access-hgqb9\") pod \"maas-api-revoke-sub-1207b689-tqskt\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.699715 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.699676 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1207b689-tqskt\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.699890 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.699751 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1207b689-tqskt\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.699938 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.699912 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgqb9\" (UniqueName: \"kubernetes.io/projected/7004e68d-f98a-4945-bf5b-2d7261c5fb37-kube-api-access-hgqb9\") pod \"maas-api-revoke-sub-1207b689-tqskt\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.700478 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.700455 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1207b689-tqskt\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.702116 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.702096 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1207b689-tqskt\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.721541 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.721511 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgqb9\" (UniqueName: \"kubernetes.io/projected/7004e68d-f98a-4945-bf5b-2d7261c5fb37-kube-api-access-hgqb9\") pod \"maas-api-revoke-sub-1207b689-tqskt\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:23.803204 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:23.803124 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:24.044333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:24.043378 4903 generic.go:356] "Generic (PLEG): container finished" podID="003f2579-8b55-4491-abdc-83140f52caba" containerID="076e3665b8ed4fcfb8f3b75ee97d0dc14fe0726d05c3bf9d94217c44a3ae868a" exitCode=0 Nov 03 22:59:24.044333 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:24.043460 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" event={"ID":"003f2579-8b55-4491-abdc-83140f52caba","Type":"ContainerDied","Data":"076e3665b8ed4fcfb8f3b75ee97d0dc14fe0726d05c3bf9d94217c44a3ae868a"} Nov 03 22:59:24.140130 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:59:24.140100 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7004e68d_f98a_4945_bf5b_2d7261c5fb37.slice/crio-11c1d39752b1fadcf6351fe30302419615411a7febbe941f7de70cc2f6953536 WatchSource:0}: Error finding container 11c1d39752b1fadcf6351fe30302419615411a7febbe941f7de70cc2f6953536: Status 404 returned error can't find the container with id 11c1d39752b1fadcf6351fe30302419615411a7febbe941f7de70cc2f6953536 Nov 03 22:59:24.142318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:24.142263 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt"] Nov 03 22:59:25.066582 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.066541 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" event={"ID":"7004e68d-f98a-4945-bf5b-2d7261c5fb37","Type":"ContainerStarted","Data":"de0b91cd23ff641b2da34d5ed61d2d97ee9ef1bb02b8fca31f361308fbccf3de"} Nov 03 22:59:25.067096 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.066602 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" event={"ID":"7004e68d-f98a-4945-bf5b-2d7261c5fb37","Type":"ContainerStarted","Data":"11c1d39752b1fadcf6351fe30302419615411a7febbe941f7de70cc2f6953536"} Nov 03 22:59:25.127168 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.127107 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" podStartSLOduration=2.127086815 podStartE2EDuration="2.127086815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:25.123276097 +0000 UTC m=+2791.527217074" watchObservedRunningTime="2026-11-03 22:59:25.127086815 +0000 UTC m=+2791.531027797" Nov 03 22:59:25.279161 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.279105 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:25.323022 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.322942 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/003f2579-8b55-4491-abdc-83140f52caba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f2579-8b55-4491-abdc-83140f52caba-maas-api-xks-ca\") pod \"003f2579-8b55-4491-abdc-83140f52caba\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " Nov 03 22:59:25.323188 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.323143 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003f2579-8b55-4491-abdc-83140f52caba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f2579-8b55-4491-abdc-83140f52caba-maas-api-openshift-ca\") pod \"003f2579-8b55-4491-abdc-83140f52caba\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " Nov 03 22:59:25.323442 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.323414 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003f2579-8b55-4491-abdc-83140f52caba-kube-api-access-jzzn2\" (UniqueName: \"kubernetes.io/projected/003f2579-8b55-4491-abdc-83140f52caba-kube-api-access-jzzn2\") pod \"003f2579-8b55-4491-abdc-83140f52caba\" (UID: \"003f2579-8b55-4491-abdc-83140f52caba\") " Nov 03 22:59:25.323626 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.323595 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003f2579-8b55-4491-abdc-83140f52caba-maas-api-openshift-ca" pod "003f2579-8b55-4491-abdc-83140f52caba" (UID: "003f2579-8b55-4491-abdc-83140f52caba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:25.324044 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.324018 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003f2579-8b55-4491-abdc-83140f52caba-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:25.325730 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.325697 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003f2579-8b55-4491-abdc-83140f52caba-kube-api-access-jzzn2" pod "003f2579-8b55-4491-abdc-83140f52caba" (UID: "003f2579-8b55-4491-abdc-83140f52caba"). InnerVolumeSpecName "kube-api-access-jzzn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:25.325839 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.325738 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003f2579-8b55-4491-abdc-83140f52caba-maas-api-xks-ca" pod "003f2579-8b55-4491-abdc-83140f52caba" (UID: "003f2579-8b55-4491-abdc-83140f52caba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:25.424948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.424916 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzzn2\" (UniqueName: \"kubernetes.io/projected/003f2579-8b55-4491-abdc-83140f52caba-kube-api-access-jzzn2\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:25.424948 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:25.424946 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003f2579-8b55-4491-abdc-83140f52caba-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:26.074857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:26.074765 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" Nov 03 22:59:26.074857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:26.074801 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26" event={"ID":"003f2579-8b55-4491-abdc-83140f52caba","Type":"ContainerDied","Data":"497640d3864036563e609c7f3289f65398d0a16bf77cb9f0234e9513a12afee1"} Nov 03 22:59:26.074857 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:26.074845 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="497640d3864036563e609c7f3289f65398d0a16bf77cb9f0234e9513a12afee1" Nov 03 22:59:26.076522 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:26.076497 4903 generic.go:356] "Generic (PLEG): container finished" podID="7004e68d-f98a-4945-bf5b-2d7261c5fb37" containerID="de0b91cd23ff641b2da34d5ed61d2d97ee9ef1bb02b8fca31f361308fbccf3de" exitCode=0 Nov 03 22:59:26.076638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:26.076539 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" event={"ID":"7004e68d-f98a-4945-bf5b-2d7261c5fb37","Type":"ContainerDied","Data":"de0b91cd23ff641b2da34d5ed61d2d97ee9ef1bb02b8fca31f361308fbccf3de"} Nov 03 22:59:27.070647 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.070615 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z"] Nov 03 22:59:27.338975 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.338912 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:27.457566 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.457530 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-xks-ca\") pod \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " Nov 03 22:59:27.457774 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.457740 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7004e68d-f98a-4945-bf5b-2d7261c5fb37-kube-api-access-hgqb9\" (UniqueName: \"kubernetes.io/projected/7004e68d-f98a-4945-bf5b-2d7261c5fb37-kube-api-access-hgqb9\") pod \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " Nov 03 22:59:27.457852 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.457798 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-openshift-ca\") pod \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\" (UID: \"7004e68d-f98a-4945-bf5b-2d7261c5fb37\") " Nov 03 22:59:27.458209 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.458179 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-openshift-ca" pod "7004e68d-f98a-4945-bf5b-2d7261c5fb37" (UID: "7004e68d-f98a-4945-bf5b-2d7261c5fb37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:27.458454 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.458438 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:27.459814 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.459791 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-xks-ca" pod "7004e68d-f98a-4945-bf5b-2d7261c5fb37" (UID: "7004e68d-f98a-4945-bf5b-2d7261c5fb37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:27.459894 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.459827 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7004e68d-f98a-4945-bf5b-2d7261c5fb37-kube-api-access-hgqb9" pod "7004e68d-f98a-4945-bf5b-2d7261c5fb37" (UID: "7004e68d-f98a-4945-bf5b-2d7261c5fb37"). InnerVolumeSpecName "kube-api-access-hgqb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:27.581320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.573257 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgqb9\" (UniqueName: \"kubernetes.io/projected/7004e68d-f98a-4945-bf5b-2d7261c5fb37-kube-api-access-hgqb9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:27.581320 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:27.573536 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7004e68d-f98a-4945-bf5b-2d7261c5fb37-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:28.093531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:28.093443 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" Nov 03 22:59:28.093531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:28.093473 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt" event={"ID":"7004e68d-f98a-4945-bf5b-2d7261c5fb37","Type":"ContainerDied","Data":"11c1d39752b1fadcf6351fe30302419615411a7febbe941f7de70cc2f6953536"} Nov 03 22:59:28.093531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:28.093527 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c1d39752b1fadcf6351fe30302419615411a7febbe941f7de70cc2f6953536" Nov 03 22:59:31.549326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.545495 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8"] Nov 03 22:59:31.549326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.546486 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7004e68d-f98a-4945-bf5b-2d7261c5fb37" containerName="revoke-keys" Nov 03 22:59:31.549326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.546505 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7004e68d-f98a-4945-bf5b-2d7261c5fb37" containerName="revoke-keys" Nov 03 22:59:31.549326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.546517 4903 cpu_manager.go:414] "containerMap: removing container" podUID="003f2579-8b55-4491-abdc-83140f52caba" containerName="revoke-keys" Nov 03 22:59:31.549326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.546523 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003f2579-8b55-4491-abdc-83140f52caba" containerName="revoke-keys" Nov 03 22:59:31.549326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.546645 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7004e68d-f98a-4945-bf5b-2d7261c5fb37" containerName="revoke-keys" Nov 03 22:59:31.549326 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.546653 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="003f2579-8b55-4491-abdc-83140f52caba" containerName="revoke-keys" Nov 03 22:59:31.556850 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.550662 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:31.571992 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.571962 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8"] Nov 03 22:59:31.575373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.574549 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ff80c0\"" Nov 03 22:59:31.575373 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.575333 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ff80c0-dockercfg-rzsmz\"" Nov 03 22:59:31.630621 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.630584 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxw96\" (UniqueName: \"kubernetes.io/projected/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-kube-api-access-gxw96\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8\" (UID: \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:31.630805 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.630781 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8\" (UID: \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:31.722413 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.722376 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s"] Nov 03 22:59:31.742316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.737433 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8\" (UID: \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:31.742316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.737766 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxw96\" (UniqueName: \"kubernetes.io/projected/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-kube-api-access-gxw96\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8\" (UID: \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:31.742316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.737818 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:31.742316 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.739598 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8\" (UID: \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:31.763336 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.763282 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxw96\" (UniqueName: \"kubernetes.io/projected/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-kube-api-access-gxw96\") pod \"payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8\" (UID: \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:31.768419 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.768393 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s"] Nov 03 22:59:31.839495 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.839407 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88nvs\" (UniqueName: \"kubernetes.io/projected/3549d146-0578-48ba-80f2-713aa4373d38-kube-api-access-88nvs\") pod \"payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s\" (UID: \"3549d146-0578-48ba-80f2-713aa4373d38\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:31.839660 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.839632 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549d146-0578-48ba-80f2-713aa4373d38-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s\" (UID: \"3549d146-0578-48ba-80f2-713aa4373d38\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:31.877345 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.877312 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:31.941697 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.941665 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549d146-0578-48ba-80f2-713aa4373d38-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s\" (UID: \"3549d146-0578-48ba-80f2-713aa4373d38\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:31.941985 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.941965 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88nvs\" (UniqueName: \"kubernetes.io/projected/3549d146-0578-48ba-80f2-713aa4373d38-kube-api-access-88nvs\") pod \"payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s\" (UID: \"3549d146-0578-48ba-80f2-713aa4373d38\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:31.942531 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.942509 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549d146-0578-48ba-80f2-713aa4373d38-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s\" (UID: \"3549d146-0578-48ba-80f2-713aa4373d38\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:31.964476 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:31.964425 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88nvs\" (UniqueName: \"kubernetes.io/projected/3549d146-0578-48ba-80f2-713aa4373d38-kube-api-access-88nvs\") pod \"payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s\" (UID: \"3549d146-0578-48ba-80f2-713aa4373d38\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:32.080339 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:32.080274 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:32.191614 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:32.191339 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8"] Nov 03 22:59:32.196102 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:59:32.196050 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5dde97b_cbf4_4441_b5fc_c8ea75e58301.slice/crio-bdb72520aef5f082d6036d2d4f328c76ab7ea6acd310542366436fea88128ab1 WatchSource:0}: Error finding container bdb72520aef5f082d6036d2d4f328c76ab7ea6acd310542366436fea88128ab1: Status 404 returned error can't find the container with id bdb72520aef5f082d6036d2d4f328c76ab7ea6acd310542366436fea88128ab1 Nov 03 22:59:32.391324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:32.391278 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s"] Nov 03 22:59:32.391718 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:59:32.391693 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3549d146_0578_48ba_80f2_713aa4373d38.slice/crio-926cb051f87cf07de1682208f1b67a3b470e8441ba8a339f827481068671a9cd WatchSource:0}: Error finding container 926cb051f87cf07de1682208f1b67a3b470e8441ba8a339f827481068671a9cd: Status 404 returned error can't find the container with id 926cb051f87cf07de1682208f1b67a3b470e8441ba8a339f827481068671a9cd Nov 03 22:59:33.140503 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:33.140466 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" event={"ID":"3549d146-0578-48ba-80f2-713aa4373d38","Type":"ContainerStarted","Data":"ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0"} Nov 03 22:59:33.140953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:33.140527 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" event={"ID":"3549d146-0578-48ba-80f2-713aa4373d38","Type":"ContainerStarted","Data":"926cb051f87cf07de1682208f1b67a3b470e8441ba8a339f827481068671a9cd"} Nov 03 22:59:33.140953 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:33.140562 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:33.142222 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:33.142198 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" event={"ID":"a5dde97b-cbf4-4441-b5fc-c8ea75e58301","Type":"ContainerStarted","Data":"eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237"} Nov 03 22:59:33.142382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:33.142234 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" event={"ID":"a5dde97b-cbf4-4441-b5fc-c8ea75e58301","Type":"ContainerStarted","Data":"bdb72520aef5f082d6036d2d4f328c76ab7ea6acd310542366436fea88128ab1"} Nov 03 22:59:33.142382 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:33.142265 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:33.179630 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:33.179573 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" podStartSLOduration=2.179555202 podStartE2EDuration="2.179555202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:33.176637638 +0000 UTC m=+2799.580578613" watchObservedRunningTime="2026-11-03 22:59:33.179555202 +0000 UTC m=+2799.583496178" Nov 03 22:59:33.216765 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:33.216714 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" podStartSLOduration=2.216701409 podStartE2EDuration="2.216701409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:33.212491619 +0000 UTC m=+2799.616432594" watchObservedRunningTime="2026-11-03 22:59:33.216701409 +0000 UTC m=+2799.620642406" Nov 03 22:59:42.092845 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.092801 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" podUID="99e7a893-2eba-47df-835c-565ebb6c1436" containerName="main" containerID="cri-o://5823963ad9d046b90701b24b9f459aaa282d1f2864ba55efb098c97b3459cc72" gracePeriod=45 Nov 03 22:59:42.114792 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.114717 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" podUID="99e7a893-2eba-47df-835c-565ebb6c1436" containerName="main" probeResult="failure" output="Get \"https://10.132.1.26:8000/ready\": dial tcp 10.132.1.26:8000: connect: connection refused" Nov 03 22:59:42.220854 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.220817 4903 generic.go:356] "Generic (PLEG): container finished" podID="99e7a893-2eba-47df-835c-565ebb6c1436" containerID="5823963ad9d046b90701b24b9f459aaa282d1f2864ba55efb098c97b3459cc72" exitCode=0 Nov 03 22:59:42.221009 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.220888 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" event={"ID":"99e7a893-2eba-47df-835c-565ebb6c1436","Type":"ContainerDied","Data":"5823963ad9d046b90701b24b9f459aaa282d1f2864ba55efb098c97b3459cc72"} Nov 03 22:59:42.400722 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.400698 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:59:42.429211 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.429175 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-tmp-dir\") pod \"99e7a893-2eba-47df-835c-565ebb6c1436\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " Nov 03 22:59:42.429432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.429263 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-dshm\") pod \"99e7a893-2eba-47df-835c-565ebb6c1436\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " Nov 03 22:59:42.429432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.429357 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-model-cache\") pod \"99e7a893-2eba-47df-835c-565ebb6c1436\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " Nov 03 22:59:42.429432 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.429395 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99e7a893-2eba-47df-835c-565ebb6c1436-tls-certs\" (UniqueName: \"kubernetes.io/secret/99e7a893-2eba-47df-835c-565ebb6c1436-tls-certs\") pod \"99e7a893-2eba-47df-835c-565ebb6c1436\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " Nov 03 22:59:42.429636 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.429496 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-tmp-dir" pod "99e7a893-2eba-47df-835c-565ebb6c1436" (UID: "99e7a893-2eba-47df-835c-565ebb6c1436"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:42.429709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.429638 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-model-cache" pod "99e7a893-2eba-47df-835c-565ebb6c1436" (UID: "99e7a893-2eba-47df-835c-565ebb6c1436"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:42.429709 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.429678 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-home\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-home\") pod \"99e7a893-2eba-47df-835c-565ebb6c1436\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " Nov 03 22:59:42.429830 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.429794 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e7a893-2eba-47df-835c-565ebb6c1436-kube-api-access-2lznc\" (UniqueName: \"kubernetes.io/projected/99e7a893-2eba-47df-835c-565ebb6c1436-kube-api-access-2lznc\") pod \"99e7a893-2eba-47df-835c-565ebb6c1436\" (UID: \"99e7a893-2eba-47df-835c-565ebb6c1436\") " Nov 03 22:59:42.429921 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.429900 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-home" pod "99e7a893-2eba-47df-835c-565ebb6c1436" (UID: "99e7a893-2eba-47df-835c-565ebb6c1436"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:42.430851 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.430823 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:42.430990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.430855 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:42.430990 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.430868 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:42.432129 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.432094 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-dshm" pod "99e7a893-2eba-47df-835c-565ebb6c1436" (UID: "99e7a893-2eba-47df-835c-565ebb6c1436"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 22:59:42.432596 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.432557 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e7a893-2eba-47df-835c-565ebb6c1436-kube-api-access-2lznc" pod "99e7a893-2eba-47df-835c-565ebb6c1436" (UID: "99e7a893-2eba-47df-835c-565ebb6c1436"). InnerVolumeSpecName "kube-api-access-2lznc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:42.432786 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.432763 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e7a893-2eba-47df-835c-565ebb6c1436-tls-certs" pod "99e7a893-2eba-47df-835c-565ebb6c1436" (UID: "99e7a893-2eba-47df-835c-565ebb6c1436"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:42.532826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.532792 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lznc\" (UniqueName: \"kubernetes.io/projected/99e7a893-2eba-47df-835c-565ebb6c1436-kube-api-access-2lznc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:42.532826 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.532824 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99e7a893-2eba-47df-835c-565ebb6c1436-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:42.533079 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:42.532842 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99e7a893-2eba-47df-835c-565ebb6c1436-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:43.230257 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:43.230231 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" Nov 03 22:59:43.230735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:43.230266 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z" event={"ID":"99e7a893-2eba-47df-835c-565ebb6c1436","Type":"ContainerDied","Data":"a82ebb80d4e77f2b7e11c3ba3e648442efa37371c390b31244a5f4862139b4b9"} Nov 03 22:59:43.230735 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:43.230346 4903 scope.go:122] "RemoveContainer" containerID="5823963ad9d046b90701b24b9f459aaa282d1f2864ba55efb098c97b3459cc72" Nov 03 22:59:43.294098 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:43.294069 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z"] Nov 03 22:59:43.310573 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:43.310540 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/sub-test-model-26f8ba-kserve-96c894f67-wzh5z"] Nov 03 22:59:43.775897 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:43.775860 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e7a893-2eba-47df-835c-565ebb6c1436" path="/var/lib/kubelet/pods/99e7a893-2eba-47df-835c-565ebb6c1436/volumes" Nov 03 22:59:44.152772 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:44.152744 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 22:59:44.153150 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:44.153122 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 22:59:46.874829 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.874791 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v"] Nov 03 22:59:46.875600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.875581 4903 cpu_manager.go:414] "containerMap: removing container" podUID="99e7a893-2eba-47df-835c-565ebb6c1436" containerName="main" Nov 03 22:59:46.875600 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.875600 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e7a893-2eba-47df-835c-565ebb6c1436" containerName="main" Nov 03 22:59:46.875761 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.875713 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e7a893-2eba-47df-835c-565ebb6c1436" containerName="main" Nov 03 22:59:46.881369 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.881344 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:46.897086 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.897057 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v"] Nov 03 22:59:46.902318 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.902265 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 22:59:46.902458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.902341 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 22:59:46.902458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.902354 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 22:59:46.902458 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:46.902353 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 22:59:47.058317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.052429 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-68410c-2444v\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.058317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.052766 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-68410c-2444v\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.058317 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.052840 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh596\" (UniqueName: \"kubernetes.io/projected/d4644b46-9129-4f5e-982f-e187da494ff5-kube-api-access-sh596\") pod \"maas-api-revoke-keys-e2e-ipp-a-68410c-2444v\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.154760 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.154726 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh596\" (UniqueName: \"kubernetes.io/projected/d4644b46-9129-4f5e-982f-e187da494ff5-kube-api-access-sh596\") pod \"maas-api-revoke-keys-e2e-ipp-a-68410c-2444v\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.154958 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.154818 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-68410c-2444v\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.155085 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.155065 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-68410c-2444v\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.155794 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.155776 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-68410c-2444v\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.157357 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.157338 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-68410c-2444v\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.176332 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.176274 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh596\" (UniqueName: \"kubernetes.io/projected/d4644b46-9129-4f5e-982f-e187da494ff5-kube-api-access-sh596\") pod \"maas-api-revoke-keys-e2e-ipp-a-68410c-2444v\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.197657 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.197626 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:47.507957 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:47.507926 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v"] Nov 03 22:59:47.508210 ip-10-0-133-193 kubenswrapper[4903]: W1103 22:59:47.508185 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4644b46_9129_4f5e_982f_e187da494ff5.slice/crio-ee8f4be5cc273e96178d6a91738020c05b54626ccabd00658baec12f11c5cf29 WatchSource:0}: Error finding container ee8f4be5cc273e96178d6a91738020c05b54626ccabd00658baec12f11c5cf29: Status 404 returned error can't find the container with id ee8f4be5cc273e96178d6a91738020c05b54626ccabd00658baec12f11c5cf29 Nov 03 22:59:48.291324 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:48.291264 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" event={"ID":"d4644b46-9129-4f5e-982f-e187da494ff5","Type":"ContainerStarted","Data":"96ccb43185b76758ee0715cfab23bc2d9ac8e0418044f33ad531fe22d6a3648b"} Nov 03 22:59:48.291751 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:48.291372 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" event={"ID":"d4644b46-9129-4f5e-982f-e187da494ff5","Type":"ContainerStarted","Data":"ee8f4be5cc273e96178d6a91738020c05b54626ccabd00658baec12f11c5cf29"} Nov 03 22:59:48.330082 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:48.330028 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" podStartSLOduration=2.330014512 podStartE2EDuration="2.330014512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 22:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 22:59:48.326944177 +0000 UTC m=+2814.730885194" watchObservedRunningTime="2026-11-03 22:59:48.330014512 +0000 UTC m=+2814.733955483" Nov 03 22:59:48.386423 ip-10-0-133-193 kubenswrapper[4903]: E1103 22:59:48.386384 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 22:59:49.300276 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:49.300239 4903 generic.go:356] "Generic (PLEG): container finished" podID="d4644b46-9129-4f5e-982f-e187da494ff5" containerID="96ccb43185b76758ee0715cfab23bc2d9ac8e0418044f33ad531fe22d6a3648b" exitCode=0 Nov 03 22:59:49.300691 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:49.300324 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" event={"ID":"d4644b46-9129-4f5e-982f-e187da494ff5","Type":"ContainerDied","Data":"96ccb43185b76758ee0715cfab23bc2d9ac8e0418044f33ad531fe22d6a3648b"} Nov 03 22:59:50.051028 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.050995 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk"] Nov 03 22:59:50.060846 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.060816 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81360fe2-9jldk"] Nov 03 22:59:50.478872 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.478844 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:50.607628 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.607538 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-openshift-ca\") pod \"d4644b46-9129-4f5e-982f-e187da494ff5\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " Nov 03 22:59:50.607791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.607658 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4644b46-9129-4f5e-982f-e187da494ff5-kube-api-access-sh596\" (UniqueName: \"kubernetes.io/projected/d4644b46-9129-4f5e-982f-e187da494ff5-kube-api-access-sh596\") pod \"d4644b46-9129-4f5e-982f-e187da494ff5\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " Nov 03 22:59:50.607791 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.607709 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-xks-ca\") pod \"d4644b46-9129-4f5e-982f-e187da494ff5\" (UID: \"d4644b46-9129-4f5e-982f-e187da494ff5\") " Nov 03 22:59:50.607968 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.607943 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-openshift-ca" pod "d4644b46-9129-4f5e-982f-e187da494ff5" (UID: "d4644b46-9129-4f5e-982f-e187da494ff5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 22:59:50.608244 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.608217 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:50.609744 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.609722 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-xks-ca" pod "d4644b46-9129-4f5e-982f-e187da494ff5" (UID: "d4644b46-9129-4f5e-982f-e187da494ff5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 22:59:50.609836 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.609797 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4644b46-9129-4f5e-982f-e187da494ff5-kube-api-access-sh596" pod "d4644b46-9129-4f5e-982f-e187da494ff5" (UID: "d4644b46-9129-4f5e-982f-e187da494ff5"). InnerVolumeSpecName "kube-api-access-sh596". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 22:59:50.710019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.709983 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh596\" (UniqueName: \"kubernetes.io/projected/d4644b46-9129-4f5e-982f-e187da494ff5-kube-api-access-sh596\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:50.710019 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:50.710015 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4644b46-9129-4f5e-982f-e187da494ff5-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 22:59:51.317810 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:51.317777 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" Nov 03 22:59:51.317983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:51.317809 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v" event={"ID":"d4644b46-9129-4f5e-982f-e187da494ff5","Type":"ContainerDied","Data":"ee8f4be5cc273e96178d6a91738020c05b54626ccabd00658baec12f11c5cf29"} Nov 03 22:59:51.317983 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:51.317875 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee8f4be5cc273e96178d6a91738020c05b54626ccabd00658baec12f11c5cf29" Nov 03 22:59:51.740274 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:51.740237 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b072383-4e99-4952-a66a-1eeb8e0bffbc" path="/var/lib/kubelet/pods/2b072383-4e99-4952-a66a-1eeb8e0bffbc/volumes" Nov 03 22:59:56.385591 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:56.385551 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 22:59:59.523617 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:59.523519 4903 scope.go:122] "RemoveContainer" containerID="090e966ee10c469a466bfc2a7f0ea02c85430cdd2e13a5dc501af02f7a174c8d" Nov 03 22:59:59.565113 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:59.564989 4903 scope.go:122] "RemoveContainer" containerID="29bbd7562a585778e30da2b62147d203f2a99f75d508b5f5151dd12a8a00d1f1" Nov 03 22:59:59.580362 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:59.580330 4903 scope.go:122] "RemoveContainer" containerID="1ba9b57d91c1044a609e35ebcb92306bd338b1f0f530e7d965f2446077478fc3" Nov 03 22:59:59.882638 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:59.882551 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2"] Nov 03 22:59:59.882944 ip-10-0-133-193 kubenswrapper[4903]: I1103 22:59:59.882891 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" podUID="28264c75-3888-44b5-ad31-e2c60d8a2dbe" containerName="discovery" containerID="cri-o://7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc" gracePeriod=30 Nov 03 23:00:00.032020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.031968 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c"] Nov 03 23:00:00.036878 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.036833 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7dbc2f7-n5j7c"] Nov 03 23:00:00.174108 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.174076 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm"] Nov 03 23:00:00.179518 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.178560 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d4644b46-9129-4f5e-982f-e187da494ff5" containerName="revoke-keys" Nov 03 23:00:00.179518 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.178587 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4644b46-9129-4f5e-982f-e187da494ff5" containerName="revoke-keys" Nov 03 23:00:00.179518 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.178879 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4644b46-9129-4f5e-982f-e187da494ff5" containerName="revoke-keys" Nov 03 23:00:00.199315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.184947 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:00.199315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.198765 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v"] Nov 03 23:00:00.234465 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.219885 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:00:00.234465 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.220643 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:00:00.234465 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.220754 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:00:00.300327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.275502 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm"] Nov 03 23:00:00.300327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.282378 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw"] Nov 03 23:00:00.300327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.288872 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz"] Nov 03 23:00:00.313472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.313430 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:00.324363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.324140 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:00.326627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.326592 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:00.403778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.382433 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9dda56-7458-454f-b28a-7b1122f5deed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm\" (UID: \"9d9dda56-7458-454f-b28a-7b1122f5deed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:00.403778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.382521 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76eb079-d83d-4467-873a-a77e020303b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm\" (UID: \"b76eb079-d83d-4467-873a-a77e020303b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:00.403778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.382562 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4hr\" (UniqueName: \"kubernetes.io/projected/9d9dda56-7458-454f-b28a-7b1122f5deed-kube-api-access-qn4hr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm\" (UID: \"9d9dda56-7458-454f-b28a-7b1122f5deed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:00.403778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.382657 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3869c86-0acc-4c84-9438-10f98b688ac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v\" (UID: \"b3869c86-0acc-4c84-9438-10f98b688ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:00.403778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.382710 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrwgw\" (UniqueName: \"kubernetes.io/projected/390914dc-6339-4110-a8f0-d3c43479e328-kube-api-access-zrwgw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw\" (UID: \"390914dc-6339-4110-a8f0-d3c43479e328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:00.403778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.382756 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390914dc-6339-4110-a8f0-d3c43479e328-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw\" (UID: \"390914dc-6339-4110-a8f0-d3c43479e328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:00.403778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.382944 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbts8\" (UniqueName: \"kubernetes.io/projected/b3869c86-0acc-4c84-9438-10f98b688ac8-kube-api-access-tbts8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v\" (UID: \"b3869c86-0acc-4c84-9438-10f98b688ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:00.403778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.383070 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77845\" (UniqueName: \"kubernetes.io/projected/b76eb079-d83d-4467-873a-a77e020303b9-kube-api-access-77845\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm\" (UID: \"b76eb079-d83d-4467-873a-a77e020303b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:00.477737 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.477693 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm"] Nov 03 23:00:00.477923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.477750 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v"] Nov 03 23:00:00.477923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.477780 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm"] Nov 03 23:00:00.478405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.478382 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:00.493098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.492854 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbts8\" (UniqueName: \"kubernetes.io/projected/b3869c86-0acc-4c84-9438-10f98b688ac8-kube-api-access-tbts8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v\" (UID: \"b3869c86-0acc-4c84-9438-10f98b688ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:00.493098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.492963 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77845\" (UniqueName: \"kubernetes.io/projected/b76eb079-d83d-4467-873a-a77e020303b9-kube-api-access-77845\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm\" (UID: \"b76eb079-d83d-4467-873a-a77e020303b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:00.493307 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.493149 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfktk\" (UniqueName: \"kubernetes.io/projected/08fcf757-4727-4a59-8e4c-b07ad317df69-kube-api-access-hfktk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz\" (UID: \"08fcf757-4727-4a59-8e4c-b07ad317df69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:00.493307 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.493187 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9dda56-7458-454f-b28a-7b1122f5deed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm\" (UID: \"9d9dda56-7458-454f-b28a-7b1122f5deed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:00.493307 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.493231 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76eb079-d83d-4467-873a-a77e020303b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm\" (UID: \"b76eb079-d83d-4467-873a-a77e020303b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:00.493307 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.493266 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4hr\" (UniqueName: \"kubernetes.io/projected/9d9dda56-7458-454f-b28a-7b1122f5deed-kube-api-access-qn4hr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm\" (UID: \"9d9dda56-7458-454f-b28a-7b1122f5deed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:00.494693 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.494658 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76eb079-d83d-4467-873a-a77e020303b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm\" (UID: \"b76eb079-d83d-4467-873a-a77e020303b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:00.494863 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.494796 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9dda56-7458-454f-b28a-7b1122f5deed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm\" (UID: \"9d9dda56-7458-454f-b28a-7b1122f5deed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:00.495641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.495498 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3869c86-0acc-4c84-9438-10f98b688ac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v\" (UID: \"b3869c86-0acc-4c84-9438-10f98b688ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:00.495641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.495544 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrwgw\" (UniqueName: \"kubernetes.io/projected/390914dc-6339-4110-a8f0-d3c43479e328-kube-api-access-zrwgw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw\" (UID: \"390914dc-6339-4110-a8f0-d3c43479e328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:00.495641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.495565 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf757-4727-4a59-8e4c-b07ad317df69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz\" (UID: \"08fcf757-4727-4a59-8e4c-b07ad317df69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:00.495641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.495583 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390914dc-6339-4110-a8f0-d3c43479e328-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw\" (UID: \"390914dc-6339-4110-a8f0-d3c43479e328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:00.497277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.497231 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3869c86-0acc-4c84-9438-10f98b688ac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v\" (UID: \"b3869c86-0acc-4c84-9438-10f98b688ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:00.497568 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.497543 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390914dc-6339-4110-a8f0-d3c43479e328-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw\" (UID: \"390914dc-6339-4110-a8f0-d3c43479e328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:00.498332 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.498177 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz"] Nov 03 23:00:00.498332 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.498238 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw"] Nov 03 23:00:00.498332 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.498266 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm"] Nov 03 23:00:00.498545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.498423 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:00.504324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.502058 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p"] Nov 03 23:00:00.518180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.517747 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4"] Nov 03 23:00:00.518786 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.518759 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4hr\" (UniqueName: \"kubernetes.io/projected/9d9dda56-7458-454f-b28a-7b1122f5deed-kube-api-access-qn4hr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm\" (UID: \"9d9dda56-7458-454f-b28a-7b1122f5deed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:00.519368 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.518978 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:00.521086 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.521063 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77845\" (UniqueName: \"kubernetes.io/projected/b76eb079-d83d-4467-873a-a77e020303b9-kube-api-access-77845\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm\" (UID: \"b76eb079-d83d-4467-873a-a77e020303b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:00.522218 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.522190 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbts8\" (UniqueName: \"kubernetes.io/projected/b3869c86-0acc-4c84-9438-10f98b688ac8-kube-api-access-tbts8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v\" (UID: \"b3869c86-0acc-4c84-9438-10f98b688ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:00.523534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.523361 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:00.524576 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.524556 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5"] Nov 03 23:00:00.525275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.525140 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:00.537858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.530040 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrwgw\" (UniqueName: \"kubernetes.io/projected/390914dc-6339-4110-a8f0-d3c43479e328-kube-api-access-zrwgw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw\" (UID: \"390914dc-6339-4110-a8f0-d3c43479e328\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:00.537858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.530221 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb"] Nov 03 23:00:00.538284 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.538263 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt"] Nov 03 23:00:00.577443 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.542584 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:00.577443 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.558453 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:00.578434 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.578396 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:00.582409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.582339 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p"] Nov 03 23:00:00.586554 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.586111 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 23:00:00.587730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.587711 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4"] Nov 03 23:00:00.594450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.593904 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz"] Nov 03 23:00:00.595188 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.595071 4903 cpu_manager.go:414] "containerMap: removing container" podUID="28264c75-3888-44b5-ad31-e2c60d8a2dbe" containerName="discovery" Nov 03 23:00:00.595188 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.595092 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28264c75-3888-44b5-ad31-e2c60d8a2dbe" containerName="discovery" Nov 03 23:00:00.595699 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.595350 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="28264c75-3888-44b5-ad31-e2c60d8a2dbe" containerName="discovery" Nov 03 23:00:00.599195 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599118 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07b85-9bec-46d6-a99c-8a703d94e303-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb\" (UID: \"c4c07b85-9bec-46d6-a99c-8a703d94e303\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599422 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxww6\" (UniqueName: \"kubernetes.io/projected/c4c07b85-9bec-46d6-a99c-8a703d94e303-kube-api-access-fxww6\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb\" (UID: \"c4c07b85-9bec-46d6-a99c-8a703d94e303\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599482 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5\" (UID: \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599510 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pqp\" (UniqueName: \"kubernetes.io/projected/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-kube-api-access-c9pqp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5\" (UID: \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599561 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba749010-2688-407c-b125-f8f88bb73af1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt\" (UID: \"ba749010-2688-407c-b125-f8f88bb73af1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599599 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p\" (UID: \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599637 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfb6\" (UniqueName: \"kubernetes.io/projected/549b99cf-bb7b-4d6e-b284-060f5a0d144d-kube-api-access-ctfb6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm\" (UID: \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599687 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5n79\" (UniqueName: \"kubernetes.io/projected/ba749010-2688-407c-b125-f8f88bb73af1-kube-api-access-q5n79\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt\" (UID: \"ba749010-2688-407c-b125-f8f88bb73af1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599805 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnfd\" (UniqueName: \"kubernetes.io/projected/a74ad0b8-94d4-4fbf-9443-d362d91bec33-kube-api-access-2dnfd\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4\" (UID: \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599915 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74ad0b8-94d4-4fbf-9443-d362d91bec33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4\" (UID: \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599941 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5"] Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.599945 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfktk\" (UniqueName: \"kubernetes.io/projected/08fcf757-4727-4a59-8e4c-b07ad317df69-kube-api-access-hfktk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz\" (UID: \"08fcf757-4727-4a59-8e4c-b07ad317df69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.600068 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/549b99cf-bb7b-4d6e-b284-060f5a0d144d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm\" (UID: \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.600161 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf757-4727-4a59-8e4c-b07ad317df69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz\" (UID: \"08fcf757-4727-4a59-8e4c-b07ad317df69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.600261 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:00.601808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.600256 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqkg\" (UniqueName: \"kubernetes.io/projected/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-kube-api-access-xhqkg\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p\" (UID: \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:00.604749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.604035 4903 generic.go:356] "Generic (PLEG): container finished" podID="28264c75-3888-44b5-ad31-e2c60d8a2dbe" containerID="7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc" exitCode=0 Nov 03 23:00:00.604749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.604159 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" Nov 03 23:00:00.604749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.604250 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" event={"ID":"28264c75-3888-44b5-ad31-e2c60d8a2dbe","Type":"ContainerDied","Data":"7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc"} Nov 03 23:00:00.604749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.604280 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2" event={"ID":"28264c75-3888-44b5-ad31-e2c60d8a2dbe","Type":"ContainerDied","Data":"5e7e133a95506d7f334270c956598f799f3949cce882fca68f3476060a3dbca6"} Nov 03 23:00:00.604749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.604332 4903 scope.go:122] "RemoveContainer" containerID="7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc" Nov 03 23:00:00.604749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.604501 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf757-4727-4a59-8e4c-b07ad317df69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz\" (UID: \"08fcf757-4727-4a59-8e4c-b07ad317df69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:00.605219 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.604872 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb"] Nov 03 23:00:00.608915 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.608897 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm"] Nov 03 23:00:00.614735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.614711 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr"] Nov 03 23:00:00.620653 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.620633 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:00.623079 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.623015 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfktk\" (UniqueName: \"kubernetes.io/projected/08fcf757-4727-4a59-8e4c-b07ad317df69-kube-api-access-hfktk\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz\" (UID: \"08fcf757-4727-4a59-8e4c-b07ad317df69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:00.623275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.623212 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz"] Nov 03 23:00:00.629141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.629124 4903 scope.go:122] "RemoveContainer" containerID="7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc" Nov 03 23:00:00.629717 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:00.629695 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc\": container with ID starting with 7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc not found: ID does not exist" containerID="7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc" Nov 03 23:00:00.629899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.629876 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc"} err="failed to get container status \"7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc\": rpc error: code = NotFound desc = could not find container \"7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc\": container with ID starting with 7bd2e1f579fc9700c68e6bb35ec5de4f437c34d49e012f552aabba356d77e3dc not found: ID does not exist" Nov 03 23:00:00.633363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.633344 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz"] Nov 03 23:00:00.633607 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.633585 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:00.634340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.634288 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr"] Nov 03 23:00:00.639561 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.639539 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb"] Nov 03 23:00:00.672597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.670524 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:00.684684 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.680513 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:00.684684 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.680683 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89"] Nov 03 23:00:00.701777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.701329 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-kubeconfig\") pod \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " Nov 03 23:00:00.701777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.701377 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-kube-api-access-ftgnd\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-kube-api-access-ftgnd\") pod \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " Nov 03 23:00:00.701777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.701418 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-token\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-token\") pod \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " Nov 03 23:00:00.701777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.701526 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-ca-configmap\") pod \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " Nov 03 23:00:00.701777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.701558 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-cacerts\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-cacerts\") pod \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " Nov 03 23:00:00.701777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.701637 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28264c75-3888-44b5-ad31-e2c60d8a2dbe-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28264c75-3888-44b5-ad31-e2c60d8a2dbe-local-certs\") pod \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " Nov 03 23:00:00.703901 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.703640 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-ca-configmap" pod "28264c75-3888-44b5-ad31-e2c60d8a2dbe" (UID: "28264c75-3888-44b5-ad31-e2c60d8a2dbe"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:00.708649 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.708612 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89"] Nov 03 23:00:00.708859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.708660 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb"] Nov 03 23:00:00.708859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.708768 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:00.712338 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.710700 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt"] Nov 03 23:00:00.713084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.712889 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-dns-cert\") pod \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\" (UID: \"28264c75-3888-44b5-ad31-e2c60d8a2dbe\") " Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.730000 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-dns-cert" pod "28264c75-3888-44b5-ad31-e2c60d8a2dbe" (UID: "28264c75-3888-44b5-ad31-e2c60d8a2dbe"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.731690 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-cacerts" pod "28264c75-3888-44b5-ad31-e2c60d8a2dbe" (UID: "28264c75-3888-44b5-ad31-e2c60d8a2dbe"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.734102 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-token" pod "28264c75-3888-44b5-ad31-e2c60d8a2dbe" (UID: "28264c75-3888-44b5-ad31-e2c60d8a2dbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736220 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ae1de-a9e1-44da-b47d-9d6177640711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz\" (UID: \"933ae1de-a9e1-44da-b47d-9d6177640711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736311 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/549b99cf-bb7b-4d6e-b284-060f5a0d144d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm\" (UID: \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736341 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b2ad97-84d0-4500-8645-172ddb9b1aa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz\" (UID: \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736405 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc69a099-02c2-45bd-aa5a-516e0680c0d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr\" (UID: \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736494 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqkg\" (UniqueName: \"kubernetes.io/projected/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-kube-api-access-xhqkg\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p\" (UID: \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736561 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07b85-9bec-46d6-a99c-8a703d94e303-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb\" (UID: \"c4c07b85-9bec-46d6-a99c-8a703d94e303\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736658 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxhv6\" (UniqueName: \"kubernetes.io/projected/e6f10255-ffed-4e46-b2b9-567ce601cc11-kube-api-access-nxhv6\") pod \"maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb\" (UID: \"e6f10255-ffed-4e46-b2b9-567ce601cc11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736691 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxww6\" (UniqueName: \"kubernetes.io/projected/c4c07b85-9bec-46d6-a99c-8a703d94e303-kube-api-access-fxww6\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb\" (UID: \"c4c07b85-9bec-46d6-a99c-8a703d94e303\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736733 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5\" (UID: \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.736785 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pqp\" (UniqueName: \"kubernetes.io/projected/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-kube-api-access-c9pqp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5\" (UID: \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737195 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba749010-2688-407c-b125-f8f88bb73af1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt\" (UID: \"ba749010-2688-407c-b125-f8f88bb73af1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737208 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/549b99cf-bb7b-4d6e-b284-060f5a0d144d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm\" (UID: \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737245 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p\" (UID: \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737404 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfb6\" (UniqueName: \"kubernetes.io/projected/549b99cf-bb7b-4d6e-b284-060f5a0d144d-kube-api-access-ctfb6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm\" (UID: \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737471 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k6s4\" (UniqueName: \"kubernetes.io/projected/55b2ad97-84d0-4500-8645-172ddb9b1aa4-kube-api-access-8k6s4\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz\" (UID: \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737510 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5n79\" (UniqueName: \"kubernetes.io/projected/ba749010-2688-407c-b125-f8f88bb73af1-kube-api-access-q5n79\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt\" (UID: \"ba749010-2688-407c-b125-f8f88bb73af1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737665 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnfd\" (UniqueName: \"kubernetes.io/projected/a74ad0b8-94d4-4fbf-9443-d362d91bec33-kube-api-access-2dnfd\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4\" (UID: \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737703 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv55g\" (UniqueName: \"kubernetes.io/projected/933ae1de-a9e1-44da-b47d-9d6177640711-kube-api-access-rv55g\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz\" (UID: \"933ae1de-a9e1-44da-b47d-9d6177640711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737817 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f10255-ffed-4e46-b2b9-567ce601cc11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb\" (UID: \"e6f10255-ffed-4e46-b2b9-567ce601cc11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737851 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrs8n\" (UniqueName: \"kubernetes.io/projected/dc69a099-02c2-45bd-aa5a-516e0680c0d2-kube-api-access-hrs8n\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr\" (UID: \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737892 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74ad0b8-94d4-4fbf-9443-d362d91bec33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4\" (UID: \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737949 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07b85-9bec-46d6-a99c-8a703d94e303-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb\" (UID: \"c4c07b85-9bec-46d6-a99c-8a703d94e303\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.737816 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5\" (UID: \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.738026 4903 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-ca-configmap\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.738038 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba749010-2688-407c-b125-f8f88bb73af1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt\" (UID: \"ba749010-2688-407c-b125-f8f88bb73af1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.738165 4903 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-cacerts\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.738192 4903 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-csr-dns-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.738209 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.738453 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28264c75-3888-44b5-ad31-e2c60d8a2dbe-local-certs" pod "28264c75-3888-44b5-ad31-e2c60d8a2dbe" (UID: "28264c75-3888-44b5-ad31-e2c60d8a2dbe"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.738658 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74ad0b8-94d4-4fbf-9443-d362d91bec33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4\" (UID: \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:00.781692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.740475 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-kube-api-access-ftgnd" pod "28264c75-3888-44b5-ad31-e2c60d8a2dbe" (UID: "28264c75-3888-44b5-ad31-e2c60d8a2dbe"). InnerVolumeSpecName "kube-api-access-ftgnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:00.794827 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.794757 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5n79\" (UniqueName: \"kubernetes.io/projected/ba749010-2688-407c-b125-f8f88bb73af1-kube-api-access-q5n79\") pod \"maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt\" (UID: \"ba749010-2688-407c-b125-f8f88bb73af1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:00.801498 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.801465 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p\" (UID: \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:00.801731 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.801709 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:00.802559 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.802533 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz"] Nov 03 23:00:00.807020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.806992 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnfd\" (UniqueName: \"kubernetes.io/projected/a74ad0b8-94d4-4fbf-9443-d362d91bec33-kube-api-access-2dnfd\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4\" (UID: \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:00.807167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.807067 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pqp\" (UniqueName: \"kubernetes.io/projected/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-kube-api-access-c9pqp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5\" (UID: \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:00.807700 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.807658 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfb6\" (UniqueName: \"kubernetes.io/projected/549b99cf-bb7b-4d6e-b284-060f5a0d144d-kube-api-access-ctfb6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm\" (UID: \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:00.807831 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.807757 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxww6\" (UniqueName: \"kubernetes.io/projected/c4c07b85-9bec-46d6-a99c-8a703d94e303-kube-api-access-fxww6\") pod \"maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb\" (UID: \"c4c07b85-9bec-46d6-a99c-8a703d94e303\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:00.808128 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.808101 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-kubeconfig" pod "28264c75-3888-44b5-ad31-e2c60d8a2dbe" (UID: "28264c75-3888-44b5-ad31-e2c60d8a2dbe"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:00.809645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.809619 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqkg\" (UniqueName: \"kubernetes.io/projected/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-kube-api-access-xhqkg\") pod \"maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p\" (UID: \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:00.817168 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.817142 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:00.865319 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.840877 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrs8n\" (UniqueName: \"kubernetes.io/projected/dc69a099-02c2-45bd-aa5a-516e0680c0d2-kube-api-access-hrs8n\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr\" (UID: \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:00.893324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.892612 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ae1de-a9e1-44da-b47d-9d6177640711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz\" (UID: \"933ae1de-a9e1-44da-b47d-9d6177640711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:00.893324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.892745 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b2ad97-84d0-4500-8645-172ddb9b1aa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz\" (UID: \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:00.893324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.892788 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc69a099-02c2-45bd-aa5a-516e0680c0d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr\" (UID: \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:00.893324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.892865 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee443a3-007b-46c1-bdea-21ab7ea099db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89\" (UID: \"bee443a3-007b-46c1-bdea-21ab7ea099db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:00.893324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.892993 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxhv6\" (UniqueName: \"kubernetes.io/projected/e6f10255-ffed-4e46-b2b9-567ce601cc11-kube-api-access-nxhv6\") pod \"maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb\" (UID: \"e6f10255-ffed-4e46-b2b9-567ce601cc11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:00.893324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.893179 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k6s4\" (UniqueName: \"kubernetes.io/projected/55b2ad97-84d0-4500-8645-172ddb9b1aa4-kube-api-access-8k6s4\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz\" (UID: \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:00.893790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.893396 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv55g\" (UniqueName: \"kubernetes.io/projected/933ae1de-a9e1-44da-b47d-9d6177640711-kube-api-access-rv55g\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz\" (UID: \"933ae1de-a9e1-44da-b47d-9d6177640711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:00.893790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.893512 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjmns\" (UniqueName: \"kubernetes.io/projected/bee443a3-007b-46c1-bdea-21ab7ea099db-kube-api-access-hjmns\") pod \"maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89\" (UID: \"bee443a3-007b-46c1-bdea-21ab7ea099db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:00.893790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.893554 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f10255-ffed-4e46-b2b9-567ce601cc11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb\" (UID: \"e6f10255-ffed-4e46-b2b9-567ce601cc11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:00.893790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.893705 4903 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28264c75-3888-44b5-ad31-e2c60d8a2dbe-istio-kubeconfig\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:00.893790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.893725 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgnd\" (UniqueName: \"kubernetes.io/projected/28264c75-3888-44b5-ad31-e2c60d8a2dbe-kube-api-access-ftgnd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:00.893790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.893744 4903 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28264c75-3888-44b5-ad31-e2c60d8a2dbe-local-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:00.907084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.894802 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f10255-ffed-4e46-b2b9-567ce601cc11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb\" (UID: \"e6f10255-ffed-4e46-b2b9-567ce601cc11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:00.907084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.877687 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:00.907084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.896235 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b2ad97-84d0-4500-8645-172ddb9b1aa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz\" (UID: \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:00.907084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.878489 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrs8n\" (UniqueName: \"kubernetes.io/projected/dc69a099-02c2-45bd-aa5a-516e0680c0d2-kube-api-access-hrs8n\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr\" (UID: \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:00.907084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.899977 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ae1de-a9e1-44da-b47d-9d6177640711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz\" (UID: \"933ae1de-a9e1-44da-b47d-9d6177640711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:00.907084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.900783 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc69a099-02c2-45bd-aa5a-516e0680c0d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr\" (UID: \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:00.914657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.914626 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:00.934091 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.932611 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:00.964440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.935441 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv55g\" (UniqueName: \"kubernetes.io/projected/933ae1de-a9e1-44da-b47d-9d6177640711-kube-api-access-rv55g\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz\" (UID: \"933ae1de-a9e1-44da-b47d-9d6177640711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:00.964440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.937366 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:00.989242 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.985664 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k6s4\" (UniqueName: \"kubernetes.io/projected/55b2ad97-84d0-4500-8645-172ddb9b1aa4-kube-api-access-8k6s4\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz\" (UID: \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:00.996255 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.995577 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjmns\" (UniqueName: \"kubernetes.io/projected/bee443a3-007b-46c1-bdea-21ab7ea099db-kube-api-access-hjmns\") pod \"maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89\" (UID: \"bee443a3-007b-46c1-bdea-21ab7ea099db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:00.996255 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.995778 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee443a3-007b-46c1-bdea-21ab7ea099db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89\" (UID: \"bee443a3-007b-46c1-bdea-21ab7ea099db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:00.996720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:00.996698 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee443a3-007b-46c1-bdea-21ab7ea099db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89\" (UID: \"bee443a3-007b-46c1-bdea-21ab7ea099db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:01.006120 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.005450 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxhv6\" (UniqueName: \"kubernetes.io/projected/e6f10255-ffed-4e46-b2b9-567ce601cc11-kube-api-access-nxhv6\") pod \"maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb\" (UID: \"e6f10255-ffed-4e46-b2b9-567ce601cc11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:01.024394 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.011660 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:01.026942 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.025998 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:01.094033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.093486 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjmns\" (UniqueName: \"kubernetes.io/projected/bee443a3-007b-46c1-bdea-21ab7ea099db-kube-api-access-hjmns\") pod \"maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89\" (UID: \"bee443a3-007b-46c1-bdea-21ab7ea099db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:01.121033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.120769 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:01.214388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.212659 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:01.217423 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.216848 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:01.221233 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.221067 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:01.223086 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.223041 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:01.281602 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.279381 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:01.295243 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.295217 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2"] Nov 03 23:00:01.301914 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.301888 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-56cz2"] Nov 03 23:00:01.380496 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.379960 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm"] Nov 03 23:00:01.677181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.677106 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" event={"ID":"9d9dda56-7458-454f-b28a-7b1122f5deed","Type":"ContainerStarted","Data":"3f9390ce4524fbae91bc3f9289aa14e65d56188d82e45b5c8b6d632f32b39cc1"} Nov 03 23:00:01.821925 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.821881 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28264c75-3888-44b5-ad31-e2c60d8a2dbe" path="/var/lib/kubelet/pods/28264c75-3888-44b5-ad31-e2c60d8a2dbe/volumes" Nov 03 23:00:01.824142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:01.824108 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c57c73-7878-4cf9-88d3-45626c062b86" path="/var/lib/kubelet/pods/d6c57c73-7878-4cf9-88d3-45626c062b86/volumes" Nov 03 23:00:02.083566 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.083535 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw"] Nov 03 23:00:02.084003 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.083975 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390914dc_6339_4110_a8f0_d3c43479e328.slice/crio-eb1e56cfab9385e996db0ce323835c0f9b0c60d73699d438ce3eac70f0c54892 WatchSource:0}: Error finding container eb1e56cfab9385e996db0ce323835c0f9b0c60d73699d438ce3eac70f0c54892: Status 404 returned error can't find the container with id eb1e56cfab9385e996db0ce323835c0f9b0c60d73699d438ce3eac70f0c54892 Nov 03 23:00:02.196151 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.196119 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm"] Nov 03 23:00:02.277472 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.277434 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3869c86_0acc_4c84_9438_10f98b688ac8.slice/crio-8554b3f02a82d4a4e2f585986707f1d775eb975e2c3f82432c52255f689969f0 WatchSource:0}: Error finding container 8554b3f02a82d4a4e2f585986707f1d775eb975e2c3f82432c52255f689969f0: Status 404 returned error can't find the container with id 8554b3f02a82d4a4e2f585986707f1d775eb975e2c3f82432c52255f689969f0 Nov 03 23:00:02.277748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.277716 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v"] Nov 03 23:00:02.488498 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.488471 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74ad0b8_94d4_4fbf_9443_d362d91bec33.slice/crio-8b0adadbc28741ae751ec2f345552f6625ff352018693cdffc7b37c047537b18 WatchSource:0}: Error finding container 8b0adadbc28741ae751ec2f345552f6625ff352018693cdffc7b37c047537b18: Status 404 returned error can't find the container with id 8b0adadbc28741ae751ec2f345552f6625ff352018693cdffc7b37c047537b18 Nov 03 23:00:02.488648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.488626 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4"] Nov 03 23:00:02.498261 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.498233 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz"] Nov 03 23:00:02.503582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.503553 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89"] Nov 03 23:00:02.504450 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.504428 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee443a3_007b_46c1_bdea_21ab7ea099db.slice/crio-4e6952cfaf300a8377579f0dfd20c214f5a3f12ba1695e139255bdb4eb7d2b3c WatchSource:0}: Error finding container 4e6952cfaf300a8377579f0dfd20c214f5a3f12ba1695e139255bdb4eb7d2b3c: Status 404 returned error can't find the container with id 4e6952cfaf300a8377579f0dfd20c214f5a3f12ba1695e139255bdb4eb7d2b3c Nov 03 23:00:02.506973 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.506942 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933ae1de_a9e1_44da_b47d_9d6177640711.slice/crio-3c02ed3bbdf0385936a908d7d1f509037e14d3e2e01442e134bfb97d18938769 WatchSource:0}: Error finding container 3c02ed3bbdf0385936a908d7d1f509037e14d3e2e01442e134bfb97d18938769: Status 404 returned error can't find the container with id 3c02ed3bbdf0385936a908d7d1f509037e14d3e2e01442e134bfb97d18938769 Nov 03 23:00:02.511226 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.511147 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm"] Nov 03 23:00:02.518599 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.518428 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr"] Nov 03 23:00:02.519118 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.518995 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549b99cf_bb7b_4d6e_b284_060f5a0d144d.slice/crio-db1f471d73e3c5cf020dd2b0192d42a12cda9fee9db310b24a783e76ee086c25 WatchSource:0}: Error finding container db1f471d73e3c5cf020dd2b0192d42a12cda9fee9db310b24a783e76ee086c25: Status 404 returned error can't find the container with id db1f471d73e3c5cf020dd2b0192d42a12cda9fee9db310b24a783e76ee086c25 Nov 03 23:00:02.525669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.525641 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz"] Nov 03 23:00:02.535368 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.535334 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc69a099_02c2_45bd_aa5a_516e0680c0d2.slice/crio-e02bd9e09df070fcd275df83b49942f4416a63bf0baa97fae13f782283c17040 WatchSource:0}: Error finding container e02bd9e09df070fcd275df83b49942f4416a63bf0baa97fae13f782283c17040: Status 404 returned error can't find the container with id e02bd9e09df070fcd275df83b49942f4416a63bf0baa97fae13f782283c17040 Nov 03 23:00:02.539995 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.539914 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b2ad97_84d0_4500_8645_172ddb9b1aa4.slice/crio-bb68e573675ddf5da15e723e885ff2abd34c922318cffb0f04c2cab7448d3d9c WatchSource:0}: Error finding container bb68e573675ddf5da15e723e885ff2abd34c922318cffb0f04c2cab7448d3d9c: Status 404 returned error can't find the container with id bb68e573675ddf5da15e723e885ff2abd34c922318cffb0f04c2cab7448d3d9c Nov 03 23:00:02.586994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.586949 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p"] Nov 03 23:00:02.611362 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.611332 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb"] Nov 03 23:00:02.615916 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.615890 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz"] Nov 03 23:00:02.619759 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.619734 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5"] Nov 03 23:00:02.623909 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.623885 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb"] Nov 03 23:00:02.628959 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.628939 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt"] Nov 03 23:00:02.689322 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.689274 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" event={"ID":"b3869c86-0acc-4c84-9438-10f98b688ac8","Type":"ContainerStarted","Data":"8554b3f02a82d4a4e2f585986707f1d775eb975e2c3f82432c52255f689969f0"} Nov 03 23:00:02.690508 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.690484 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" event={"ID":"549b99cf-bb7b-4d6e-b284-060f5a0d144d","Type":"ContainerStarted","Data":"db1f471d73e3c5cf020dd2b0192d42a12cda9fee9db310b24a783e76ee086c25"} Nov 03 23:00:02.691587 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.691565 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" event={"ID":"bee443a3-007b-46c1-bdea-21ab7ea099db","Type":"ContainerStarted","Data":"4e6952cfaf300a8377579f0dfd20c214f5a3f12ba1695e139255bdb4eb7d2b3c"} Nov 03 23:00:02.692839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.692818 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" event={"ID":"b76eb079-d83d-4467-873a-a77e020303b9","Type":"ContainerStarted","Data":"be237537d3917f606dd7254b095ccde85c577fffeedb1a23c037d3dd7a60dff4"} Nov 03 23:00:02.694250 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.694229 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" event={"ID":"dc69a099-02c2-45bd-aa5a-516e0680c0d2","Type":"ContainerStarted","Data":"e02bd9e09df070fcd275df83b49942f4416a63bf0baa97fae13f782283c17040"} Nov 03 23:00:02.696500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.696478 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" event={"ID":"55b2ad97-84d0-4500-8645-172ddb9b1aa4","Type":"ContainerStarted","Data":"bb68e573675ddf5da15e723e885ff2abd34c922318cffb0f04c2cab7448d3d9c"} Nov 03 23:00:02.697806 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.697783 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" event={"ID":"a74ad0b8-94d4-4fbf-9443-d362d91bec33","Type":"ContainerStarted","Data":"8b0adadbc28741ae751ec2f345552f6625ff352018693cdffc7b37c047537b18"} Nov 03 23:00:02.699067 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.699038 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" event={"ID":"933ae1de-a9e1-44da-b47d-9d6177640711","Type":"ContainerStarted","Data":"3c02ed3bbdf0385936a908d7d1f509037e14d3e2e01442e134bfb97d18938769"} Nov 03 23:00:02.700564 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.700546 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" event={"ID":"9d9dda56-7458-454f-b28a-7b1122f5deed","Type":"ContainerStarted","Data":"a34ba4c1088cc27531251e8d52f8185ed621542f31f6d40b61190c497d070764"} Nov 03 23:00:02.701949 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.701927 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" event={"ID":"390914dc-6339-4110-a8f0-d3c43479e328","Type":"ContainerStarted","Data":"eb1e56cfab9385e996db0ce323835c0f9b0c60d73699d438ce3eac70f0c54892"} Nov 03 23:00:02.706944 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.706922 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f11ee4_b643_4fa5_b040_6e3532ee98a3.slice/crio-f93ee5063fb3da6ca60ee3e38991d2097ed4232055393042b8e67a0d980e6396 WatchSource:0}: Error finding container f93ee5063fb3da6ca60ee3e38991d2097ed4232055393042b8e67a0d980e6396: Status 404 returned error can't find the container with id f93ee5063fb3da6ca60ee3e38991d2097ed4232055393042b8e67a0d980e6396 Nov 03 23:00:02.739929 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:02.739819 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" podStartSLOduration=2.739802023 podStartE2EDuration="2.739802023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:02.737257798 +0000 UTC m=+2829.141198773" watchObservedRunningTime="2026-11-03 23:00:02.739802023 +0000 UTC m=+2829.143743000" Nov 03 23:00:02.780781 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.780499 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fcf757_4727_4a59_8e4c_b07ad317df69.slice/crio-4b1ddf2be88ef323aec8d2abaf01e3310f89e832db5453562919ca6a23e64b7f WatchSource:0}: Error finding container 4b1ddf2be88ef323aec8d2abaf01e3310f89e832db5453562919ca6a23e64b7f: Status 404 returned error can't find the container with id 4b1ddf2be88ef323aec8d2abaf01e3310f89e832db5453562919ca6a23e64b7f Nov 03 23:00:02.781713 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.781691 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c07b85_9bec_46d6_a99c_8a703d94e303.slice/crio-7768fe2d83b0ef0aa77cedea84dd8530c3d8be9d143d1d97b47a3c72162766ba WatchSource:0}: Error finding container 7768fe2d83b0ef0aa77cedea84dd8530c3d8be9d143d1d97b47a3c72162766ba: Status 404 returned error can't find the container with id 7768fe2d83b0ef0aa77cedea84dd8530c3d8be9d143d1d97b47a3c72162766ba Nov 03 23:00:02.787661 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.787636 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a165c7_65a1_4e2d_9b8b_10d962afbf1b.slice/crio-d9ef5fd51c12f942ac7b87f6ed8249afa52a8c902895a1789dbc07321dedaa5b WatchSource:0}: Error finding container d9ef5fd51c12f942ac7b87f6ed8249afa52a8c902895a1789dbc07321dedaa5b: Status 404 returned error can't find the container with id d9ef5fd51c12f942ac7b87f6ed8249afa52a8c902895a1789dbc07321dedaa5b Nov 03 23:00:02.790070 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.790046 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f10255_ffed_4e46_b2b9_567ce601cc11.slice/crio-af211dcdd35243578f3b099fc23201b1726c1105b98e9c9b9c6edbbc1e70880d WatchSource:0}: Error finding container af211dcdd35243578f3b099fc23201b1726c1105b98e9c9b9c6edbbc1e70880d: Status 404 returned error can't find the container with id af211dcdd35243578f3b099fc23201b1726c1105b98e9c9b9c6edbbc1e70880d Nov 03 23:00:02.792486 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:02.792467 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba749010_2688_407c_b125_f8f88bb73af1.slice/crio-57585a47c736f6ff67a2aeee2dda22732c998e1f46ac3e7ed45a62b7e6408cc4 WatchSource:0}: Error finding container 57585a47c736f6ff67a2aeee2dda22732c998e1f46ac3e7ed45a62b7e6408cc4: Status 404 returned error can't find the container with id 57585a47c736f6ff67a2aeee2dda22732c998e1f46ac3e7ed45a62b7e6408cc4 Nov 03 23:00:03.711991 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.711950 4903 generic.go:356] "Generic (PLEG): container finished" podID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerID="e96aa0c5d23c9ac69019da186783f5ee7df6e810353bf45c8d91f9904f0197eb" exitCode=60 Nov 03 23:00:03.712424 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.712028 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" event={"ID":"b3869c86-0acc-4c84-9438-10f98b688ac8","Type":"ContainerDied","Data":"e96aa0c5d23c9ac69019da186783f5ee7df6e810353bf45c8d91f9904f0197eb"} Nov 03 23:00:03.712512 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.712494 4903 scope.go:122] "RemoveContainer" containerID="e96aa0c5d23c9ac69019da186783f5ee7df6e810353bf45c8d91f9904f0197eb" Nov 03 23:00:03.715648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.715393 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" event={"ID":"08fcf757-4727-4a59-8e4c-b07ad317df69","Type":"ContainerStarted","Data":"4b1ddf2be88ef323aec8d2abaf01e3310f89e832db5453562919ca6a23e64b7f"} Nov 03 23:00:03.721601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.716985 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" event={"ID":"549b99cf-bb7b-4d6e-b284-060f5a0d144d","Type":"ContainerStarted","Data":"3b4e824e99c054eb7ba073354ddda76d3f74e1f25a8518bf134bb7261faba4c8"} Nov 03 23:00:03.721601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.719913 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" event={"ID":"bee443a3-007b-46c1-bdea-21ab7ea099db","Type":"ContainerStarted","Data":"492590ade3179f808e0b9eb11b70f23f080f479e97b65ad444f1102e1097dbc4"} Nov 03 23:00:03.722458 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.722428 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" event={"ID":"42a165c7-65a1-4e2d-9b8b-10d962afbf1b","Type":"ContainerStarted","Data":"d9ef5fd51c12f942ac7b87f6ed8249afa52a8c902895a1789dbc07321dedaa5b"} Nov 03 23:00:03.724648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.724625 4903 generic.go:356] "Generic (PLEG): container finished" podID="b76eb079-d83d-4467-873a-a77e020303b9" containerID="5f80635314513674d11360bec223be2700be5031cabf7fe0673834e73aad7b48" exitCode=60 Nov 03 23:00:03.724937 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.724729 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" event={"ID":"b76eb079-d83d-4467-873a-a77e020303b9","Type":"ContainerDied","Data":"5f80635314513674d11360bec223be2700be5031cabf7fe0673834e73aad7b48"} Nov 03 23:00:03.725060 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.725028 4903 scope.go:122] "RemoveContainer" containerID="5f80635314513674d11360bec223be2700be5031cabf7fe0673834e73aad7b48" Nov 03 23:00:03.741753 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.741146 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" event={"ID":"dc69a099-02c2-45bd-aa5a-516e0680c0d2","Type":"ContainerStarted","Data":"ae5d5d821663e0b1e8c4a922a940445fe97d2ba0a4c66bd27ae2dd00e1068107"} Nov 03 23:00:03.741753 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.741206 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" event={"ID":"ba749010-2688-407c-b125-f8f88bb73af1","Type":"ContainerStarted","Data":"57585a47c736f6ff67a2aeee2dda22732c998e1f46ac3e7ed45a62b7e6408cc4"} Nov 03 23:00:03.741753 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.741236 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" event={"ID":"e6f10255-ffed-4e46-b2b9-567ce601cc11","Type":"ContainerStarted","Data":"af211dcdd35243578f3b099fc23201b1726c1105b98e9c9b9c6edbbc1e70880d"} Nov 03 23:00:03.777026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.776451 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" event={"ID":"c4c07b85-9bec-46d6-a99c-8a703d94e303","Type":"ContainerStarted","Data":"7768fe2d83b0ef0aa77cedea84dd8530c3d8be9d143d1d97b47a3c72162766ba"} Nov 03 23:00:03.779745 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.778821 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" event={"ID":"d7f11ee4-b643-4fa5-b040-6e3532ee98a3","Type":"ContainerStarted","Data":"f93ee5063fb3da6ca60ee3e38991d2097ed4232055393042b8e67a0d980e6396"} Nov 03 23:00:03.781418 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.781388 4903 generic.go:356] "Generic (PLEG): container finished" podID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerID="2092f7e092ac6337a770a2ae2fa4c5bd13bbe76e94987593cd6872b40265e621" exitCode=60 Nov 03 23:00:03.781532 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.781484 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" event={"ID":"a74ad0b8-94d4-4fbf-9443-d362d91bec33","Type":"ContainerDied","Data":"2092f7e092ac6337a770a2ae2fa4c5bd13bbe76e94987593cd6872b40265e621"} Nov 03 23:00:03.781876 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.781855 4903 scope.go:122] "RemoveContainer" containerID="2092f7e092ac6337a770a2ae2fa4c5bd13bbe76e94987593cd6872b40265e621" Nov 03 23:00:03.784213 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.784187 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" event={"ID":"933ae1de-a9e1-44da-b47d-9d6177640711","Type":"ContainerStarted","Data":"f62fa6154d9ffe4bb5b6a8963f561aa005cdba61b677aa2870c9a53276de4723"} Nov 03 23:00:03.791648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.791618 4903 generic.go:356] "Generic (PLEG): container finished" podID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerID="a34ba4c1088cc27531251e8d52f8185ed621542f31f6d40b61190c497d070764" exitCode=60 Nov 03 23:00:03.791762 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.791714 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" event={"ID":"9d9dda56-7458-454f-b28a-7b1122f5deed","Type":"ContainerDied","Data":"a34ba4c1088cc27531251e8d52f8185ed621542f31f6d40b61190c497d070764"} Nov 03 23:00:03.792218 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.791951 4903 scope.go:122] "RemoveContainer" containerID="a34ba4c1088cc27531251e8d52f8185ed621542f31f6d40b61190c497d070764" Nov 03 23:00:03.798033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.798010 4903 generic.go:356] "Generic (PLEG): container finished" podID="390914dc-6339-4110-a8f0-d3c43479e328" containerID="920f82066aa07fd77c49d5ee020df4c4510261ffb0a9a08f7403efae1bffcc81" exitCode=60 Nov 03 23:00:03.798149 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.798085 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" event={"ID":"390914dc-6339-4110-a8f0-d3c43479e328","Type":"ContainerDied","Data":"920f82066aa07fd77c49d5ee020df4c4510261ffb0a9a08f7403efae1bffcc81"} Nov 03 23:00:03.798440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.798415 4903 scope.go:122] "RemoveContainer" containerID="920f82066aa07fd77c49d5ee020df4c4510261ffb0a9a08f7403efae1bffcc81" Nov 03 23:00:03.893781 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.893715 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" podStartSLOduration=3.893254528 podStartE2EDuration="3.893254528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:03.887845107 +0000 UTC m=+2830.291786084" watchObservedRunningTime="2026-11-03 23:00:03.893254528 +0000 UTC m=+2830.297195504" Nov 03 23:00:03.930138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.930034 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" podStartSLOduration=3.930019286 podStartE2EDuration="3.930019286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:03.92552569 +0000 UTC m=+2830.329466665" watchObservedRunningTime="2026-11-03 23:00:03.930019286 +0000 UTC m=+2830.333960308" Nov 03 23:00:03.983964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:03.983924 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" podStartSLOduration=3.9839098379999998 podStartE2EDuration="3.983909838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:03.977553828 +0000 UTC m=+2830.381494830" watchObservedRunningTime="2026-11-03 23:00:03.983909838 +0000 UTC m=+2830.387850813" Nov 03 23:00:04.014955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.014902 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" podStartSLOduration=4.014889262 podStartE2EDuration="4.014889262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:04.013436802 +0000 UTC m=+2830.417377778" watchObservedRunningTime="2026-11-03 23:00:04.014889262 +0000 UTC m=+2830.418830237" Nov 03 23:00:04.696538 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.696494 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-9b67b1"] Nov 03 23:00:04.702003 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.701935 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-9b67b1" Nov 03 23:00:04.716117 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.716088 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-9b67b1"] Nov 03 23:00:04.716492 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.716344 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-khmwt\"" Nov 03 23:00:04.720702 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.720676 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzm4d\" (UniqueName: \"kubernetes.io/projected/eeea961c-52d6-4cf9-87b8-10a7b1dad6b0-kube-api-access-qzm4d\") pod \"test-curl-7257-9b67b1\" (UID: \"eeea961c-52d6-4cf9-87b8-10a7b1dad6b0\") " pod="opendatahub/test-curl-7257-9b67b1" Nov 03 23:00:04.809116 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.809082 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" event={"ID":"ba749010-2688-407c-b125-f8f88bb73af1","Type":"ContainerStarted","Data":"4107a32bba11d0de583a4bdfa2178d87f0ed6be0934ddf8b9b8e884e2829a0fe"} Nov 03 23:00:04.810964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.810936 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" event={"ID":"e6f10255-ffed-4e46-b2b9-567ce601cc11","Type":"ContainerStarted","Data":"8b1a91f54335b81c13910f6bee7c80d760d9b895b11f1b58d18f3fff7487e7bd"} Nov 03 23:00:04.812619 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.812592 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" event={"ID":"c4c07b85-9bec-46d6-a99c-8a703d94e303","Type":"ContainerStarted","Data":"c2d0dec49b25fd9d9c106df9237bd65ac3a280d2ac7798c32d52070075058ce9"} Nov 03 23:00:04.814223 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.814195 4903 generic.go:356] "Generic (PLEG): container finished" podID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerID="26d99b868aed98b4c6ecca6c6a988e541671f01a0b4261eee680ce98472b27c7" exitCode=60 Nov 03 23:00:04.814325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.814261 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" event={"ID":"d7f11ee4-b643-4fa5-b040-6e3532ee98a3","Type":"ContainerDied","Data":"26d99b868aed98b4c6ecca6c6a988e541671f01a0b4261eee680ce98472b27c7"} Nov 03 23:00:04.814507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.814488 4903 scope.go:122] "RemoveContainer" containerID="26d99b868aed98b4c6ecca6c6a988e541671f01a0b4261eee680ce98472b27c7" Nov 03 23:00:04.816366 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.816339 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" event={"ID":"a74ad0b8-94d4-4fbf-9443-d362d91bec33","Type":"ContainerStarted","Data":"7db9e6de64351ba54f67b4f313baa153db4f670a4fff5d4eed8b3e3cdb99ddbc"} Nov 03 23:00:04.817999 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.817954 4903 generic.go:356] "Generic (PLEG): container finished" podID="933ae1de-a9e1-44da-b47d-9d6177640711" containerID="f62fa6154d9ffe4bb5b6a8963f561aa005cdba61b677aa2870c9a53276de4723" exitCode=60 Nov 03 23:00:04.818059 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.818037 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" event={"ID":"933ae1de-a9e1-44da-b47d-9d6177640711","Type":"ContainerDied","Data":"f62fa6154d9ffe4bb5b6a8963f561aa005cdba61b677aa2870c9a53276de4723"} Nov 03 23:00:04.818220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.818203 4903 scope.go:122] "RemoveContainer" containerID="f62fa6154d9ffe4bb5b6a8963f561aa005cdba61b677aa2870c9a53276de4723" Nov 03 23:00:04.820075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.820054 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" event={"ID":"b3869c86-0acc-4c84-9438-10f98b688ac8","Type":"ContainerStarted","Data":"6d9ebfaa16f6ded18409b177ff30da2e31e56a745b07d4e4cfedeb7a2c9a6ece"} Nov 03 23:00:04.821503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.821483 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzm4d\" (UniqueName: \"kubernetes.io/projected/eeea961c-52d6-4cf9-87b8-10a7b1dad6b0-kube-api-access-qzm4d\") pod \"test-curl-7257-9b67b1\" (UID: \"eeea961c-52d6-4cf9-87b8-10a7b1dad6b0\") " pod="opendatahub/test-curl-7257-9b67b1" Nov 03 23:00:04.821767 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.821737 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" event={"ID":"08fcf757-4727-4a59-8e4c-b07ad317df69","Type":"ContainerStarted","Data":"4f760807dfb39b3d25a93f0d00c8e527034a89c51772993d8b93154e015b4bd4"} Nov 03 23:00:04.823451 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.823421 4903 generic.go:356] "Generic (PLEG): container finished" podID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerID="3b4e824e99c054eb7ba073354ddda76d3f74e1f25a8518bf134bb7261faba4c8" exitCode=60 Nov 03 23:00:04.823568 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.823489 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" event={"ID":"549b99cf-bb7b-4d6e-b284-060f5a0d144d","Type":"ContainerDied","Data":"3b4e824e99c054eb7ba073354ddda76d3f74e1f25a8518bf134bb7261faba4c8"} Nov 03 23:00:04.824100 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.823731 4903 scope.go:122] "RemoveContainer" containerID="3b4e824e99c054eb7ba073354ddda76d3f74e1f25a8518bf134bb7261faba4c8" Nov 03 23:00:04.833004 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.832817 4903 generic.go:356] "Generic (PLEG): container finished" podID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerID="492590ade3179f808e0b9eb11b70f23f080f479e97b65ad444f1102e1097dbc4" exitCode=60 Nov 03 23:00:04.833004 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.832899 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" event={"ID":"bee443a3-007b-46c1-bdea-21ab7ea099db","Type":"ContainerDied","Data":"492590ade3179f808e0b9eb11b70f23f080f479e97b65ad444f1102e1097dbc4"} Nov 03 23:00:04.833268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.833248 4903 scope.go:122] "RemoveContainer" containerID="492590ade3179f808e0b9eb11b70f23f080f479e97b65ad444f1102e1097dbc4" Nov 03 23:00:04.834998 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.834972 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" event={"ID":"42a165c7-65a1-4e2d-9b8b-10d962afbf1b","Type":"ContainerStarted","Data":"acbf5f9bc9ac06ea8a9b9a24eb2358ea102ddd6c29bbe3dd332b985f7c6a9ec2"} Nov 03 23:00:04.838070 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.838045 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" event={"ID":"b76eb079-d83d-4467-873a-a77e020303b9","Type":"ContainerStarted","Data":"3ca9b7952bd2dd749803274d9af2021e7e78b835217d747cc3162bf08021c22f"} Nov 03 23:00:04.840024 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.839996 4903 generic.go:356] "Generic (PLEG): container finished" podID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerID="ae5d5d821663e0b1e8c4a922a940445fe97d2ba0a4c66bd27ae2dd00e1068107" exitCode=60 Nov 03 23:00:04.840110 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.840071 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" event={"ID":"dc69a099-02c2-45bd-aa5a-516e0680c0d2","Type":"ContainerDied","Data":"ae5d5d821663e0b1e8c4a922a940445fe97d2ba0a4c66bd27ae2dd00e1068107"} Nov 03 23:00:04.840275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.840250 4903 scope.go:122] "RemoveContainer" containerID="ae5d5d821663e0b1e8c4a922a940445fe97d2ba0a4c66bd27ae2dd00e1068107" Nov 03 23:00:04.855350 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.849957 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" podStartSLOduration=4.849937607 podStartE2EDuration="4.849937607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:04.848327847 +0000 UTC m=+2831.252268823" watchObservedRunningTime="2026-11-03 23:00:04.849937607 +0000 UTC m=+2831.253878583" Nov 03 23:00:04.855350 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.850752 4903 generic.go:356] "Generic (PLEG): container finished" podID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerID="8235dfd97a1877afef764f0fc00a3e700a8d12b83007aa290eb773e89b89919d" exitCode=60 Nov 03 23:00:04.855350 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.850791 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" event={"ID":"55b2ad97-84d0-4500-8645-172ddb9b1aa4","Type":"ContainerDied","Data":"8235dfd97a1877afef764f0fc00a3e700a8d12b83007aa290eb773e89b89919d"} Nov 03 23:00:04.855350 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.851271 4903 scope.go:122] "RemoveContainer" containerID="8235dfd97a1877afef764f0fc00a3e700a8d12b83007aa290eb773e89b89919d" Nov 03 23:00:04.856640 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.856612 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzm4d\" (UniqueName: \"kubernetes.io/projected/eeea961c-52d6-4cf9-87b8-10a7b1dad6b0-kube-api-access-qzm4d\") pod \"test-curl-7257-9b67b1\" (UID: \"eeea961c-52d6-4cf9-87b8-10a7b1dad6b0\") " pod="opendatahub/test-curl-7257-9b67b1" Nov 03 23:00:04.915182 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.915129 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" podStartSLOduration=4.915111383 podStartE2EDuration="4.915111383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:04.906546697 +0000 UTC m=+2831.310487672" watchObservedRunningTime="2026-11-03 23:00:04.915111383 +0000 UTC m=+2831.319052361" Nov 03 23:00:04.974331 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:04.970383 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-9b67b1" Nov 03 23:00:05.003262 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.003211 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" podStartSLOduration=5.003195863 podStartE2EDuration="5.003195863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:05.000951057 +0000 UTC m=+2831.404892028" watchObservedRunningTime="2026-11-03 23:00:05.003195863 +0000 UTC m=+2831.407136838" Nov 03 23:00:05.043016 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.040924 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" podStartSLOduration=5.04090382 podStartE2EDuration="5.04090382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:05.038089409 +0000 UTC m=+2831.442030384" watchObservedRunningTime="2026-11-03 23:00:05.04090382 +0000 UTC m=+2831.444844798" Nov 03 23:00:05.083499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.080533 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" podStartSLOduration=5.080516415 podStartE2EDuration="5.080516415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:05.0760004 +0000 UTC m=+2831.479941376" watchObservedRunningTime="2026-11-03 23:00:05.080516415 +0000 UTC m=+2831.484457400" Nov 03 23:00:05.186969 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.186931 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht"] Nov 03 23:00:05.192209 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.192176 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:05.210458 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.210429 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a4b994\"" Nov 03 23:00:05.210993 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.210970 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a4b994-dockercfg-2qr2c\"" Nov 03 23:00:05.213268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.213240 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht"] Nov 03 23:00:05.230013 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.229978 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8f6\" (UniqueName: \"kubernetes.io/projected/7423e5ac-0e71-463a-b099-f7af3f31a1d8-kube-api-access-zx8f6\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht\" (UID: \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:05.230212 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.230043 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423e5ac-0e71-463a-b099-f7af3f31a1d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht\" (UID: \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:05.247182 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.246724 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" podStartSLOduration=5.246704445 podStartE2EDuration="5.246704445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:05.245943375 +0000 UTC m=+2831.649884350" watchObservedRunningTime="2026-11-03 23:00:05.246704445 +0000 UTC m=+2831.650645420" Nov 03 23:00:05.306572 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.301087 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5"] Nov 03 23:00:05.323634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.323605 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5"] Nov 03 23:00:05.324765 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.323916 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:05.333513 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.331565 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8f6\" (UniqueName: \"kubernetes.io/projected/7423e5ac-0e71-463a-b099-f7af3f31a1d8-kube-api-access-zx8f6\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht\" (UID: \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:05.333513 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.331658 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423e5ac-0e71-463a-b099-f7af3f31a1d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht\" (UID: \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:05.333513 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.332868 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423e5ac-0e71-463a-b099-f7af3f31a1d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht\" (UID: \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:05.402571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.401983 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8f6\" (UniqueName: \"kubernetes.io/projected/7423e5ac-0e71-463a-b099-f7af3f31a1d8-kube-api-access-zx8f6\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht\" (UID: \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:05.416946 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.416883 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" podStartSLOduration=5.416863141 podStartE2EDuration="5.416863141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:05.402150025 +0000 UTC m=+2831.806090992" watchObservedRunningTime="2026-11-03 23:00:05.416863141 +0000 UTC m=+2831.820804117" Nov 03 23:00:05.433367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.433250 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5\" (UID: \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:05.433539 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.433421 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq8h7\" (UniqueName: \"kubernetes.io/projected/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-kube-api-access-pq8h7\") pod \"payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5\" (UID: \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:05.492730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.492666 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" podStartSLOduration=5.492646019 podStartE2EDuration="5.492646019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:05.441710857 +0000 UTC m=+2831.845651833" watchObservedRunningTime="2026-11-03 23:00:05.492646019 +0000 UTC m=+2831.896586995" Nov 03 23:00:05.535474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.535068 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5\" (UID: \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:05.535474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.535200 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq8h7\" (UniqueName: \"kubernetes.io/projected/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-kube-api-access-pq8h7\") pod \"payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5\" (UID: \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:05.536498 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.536467 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5\" (UID: \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:05.557390 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.557356 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq8h7\" (UniqueName: \"kubernetes.io/projected/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-kube-api-access-pq8h7\") pod \"payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5\" (UID: \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:05.693689 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.693656 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-9b67b1"] Nov 03 23:00:05.865310 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.865266 4903 generic.go:356] "Generic (PLEG): container finished" podID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerID="7e3eda9b1a703b437c64eb2ce72ec589560bf29b809c43f9d79cd431fa6ee25e" exitCode=60 Nov 03 23:00:05.865790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.865329 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" event={"ID":"9d9dda56-7458-454f-b28a-7b1122f5deed","Type":"ContainerDied","Data":"7e3eda9b1a703b437c64eb2ce72ec589560bf29b809c43f9d79cd431fa6ee25e"} Nov 03 23:00:05.865790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.865398 4903 scope.go:122] "RemoveContainer" containerID="a34ba4c1088cc27531251e8d52f8185ed621542f31f6d40b61190c497d070764" Nov 03 23:00:05.865790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.865751 4903 scope.go:122] "RemoveContainer" containerID="7e3eda9b1a703b437c64eb2ce72ec589560bf29b809c43f9d79cd431fa6ee25e" Nov 03 23:00:05.866192 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:05.866164 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm_odh-ai-gateway-infra(9d9dda56-7458-454f-b28a-7b1122f5deed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" Nov 03 23:00:05.874143 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.874118 4903 generic.go:356] "Generic (PLEG): container finished" podID="390914dc-6339-4110-a8f0-d3c43479e328" containerID="403465c115220ce049b9fb91255e3bd10c57313fe9d9fecd363789737c1abbc8" exitCode=60 Nov 03 23:00:05.874273 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.874222 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" event={"ID":"390914dc-6339-4110-a8f0-d3c43479e328","Type":"ContainerDied","Data":"403465c115220ce049b9fb91255e3bd10c57313fe9d9fecd363789737c1abbc8"} Nov 03 23:00:05.874747 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.874729 4903 scope.go:122] "RemoveContainer" containerID="403465c115220ce049b9fb91255e3bd10c57313fe9d9fecd363789737c1abbc8" Nov 03 23:00:05.875079 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:05.875061 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw_odh-ai-gateway-infra(390914dc-6339-4110-a8f0-d3c43479e328)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" podUID="390914dc-6339-4110-a8f0-d3c43479e328" Nov 03 23:00:05.886706 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.886664 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:05.889119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.889097 4903 scope.go:122] "RemoveContainer" containerID="920f82066aa07fd77c49d5ee020df4c4510261ffb0a9a08f7403efae1bffcc81" Nov 03 23:00:05.890202 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.890071 4903 generic.go:356] "Generic (PLEG): container finished" podID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerID="6d9ebfaa16f6ded18409b177ff30da2e31e56a745b07d4e4cfedeb7a2c9a6ece" exitCode=60 Nov 03 23:00:05.890202 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.890162 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" event={"ID":"b3869c86-0acc-4c84-9438-10f98b688ac8","Type":"ContainerDied","Data":"6d9ebfaa16f6ded18409b177ff30da2e31e56a745b07d4e4cfedeb7a2c9a6ece"} Nov 03 23:00:05.890366 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.890217 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:05.890579 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.890555 4903 scope.go:122] "RemoveContainer" containerID="6d9ebfaa16f6ded18409b177ff30da2e31e56a745b07d4e4cfedeb7a2c9a6ece" Nov 03 23:00:05.890962 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:05.890939 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v_odh-ai-gateway-infra(b3869c86-0acc-4c84-9438-10f98b688ac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" Nov 03 23:00:05.892539 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.892520 4903 generic.go:356] "Generic (PLEG): container finished" podID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerID="4f760807dfb39b3d25a93f0d00c8e527034a89c51772993d8b93154e015b4bd4" exitCode=60 Nov 03 23:00:05.892695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.892575 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" event={"ID":"08fcf757-4727-4a59-8e4c-b07ad317df69","Type":"ContainerDied","Data":"4f760807dfb39b3d25a93f0d00c8e527034a89c51772993d8b93154e015b4bd4"} Nov 03 23:00:05.892885 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.892868 4903 scope.go:122] "RemoveContainer" containerID="4f760807dfb39b3d25a93f0d00c8e527034a89c51772993d8b93154e015b4bd4" Nov 03 23:00:05.894907 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.894886 4903 generic.go:356] "Generic (PLEG): container finished" podID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerID="acbf5f9bc9ac06ea8a9b9a24eb2358ea102ddd6c29bbe3dd332b985f7c6a9ec2" exitCode=60 Nov 03 23:00:05.895043 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.894975 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" event={"ID":"42a165c7-65a1-4e2d-9b8b-10d962afbf1b","Type":"ContainerDied","Data":"acbf5f9bc9ac06ea8a9b9a24eb2358ea102ddd6c29bbe3dd332b985f7c6a9ec2"} Nov 03 23:00:05.895195 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.895176 4903 scope.go:122] "RemoveContainer" containerID="acbf5f9bc9ac06ea8a9b9a24eb2358ea102ddd6c29bbe3dd332b985f7c6a9ec2" Nov 03 23:00:05.897242 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.897218 4903 generic.go:356] "Generic (PLEG): container finished" podID="b76eb079-d83d-4467-873a-a77e020303b9" containerID="3ca9b7952bd2dd749803274d9af2021e7e78b835217d747cc3162bf08021c22f" exitCode=60 Nov 03 23:00:05.897386 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.897338 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" event={"ID":"b76eb079-d83d-4467-873a-a77e020303b9","Type":"ContainerDied","Data":"3ca9b7952bd2dd749803274d9af2021e7e78b835217d747cc3162bf08021c22f"} Nov 03 23:00:05.897574 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.897560 4903 scope.go:122] "RemoveContainer" containerID="3ca9b7952bd2dd749803274d9af2021e7e78b835217d747cc3162bf08021c22f" Nov 03 23:00:05.897788 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:05.897771 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm_odh-ai-gateway-infra(b76eb079-d83d-4467-873a-a77e020303b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" podUID="b76eb079-d83d-4467-873a-a77e020303b9" Nov 03 23:00:05.899702 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.899682 4903 generic.go:356] "Generic (PLEG): container finished" podID="ba749010-2688-407c-b125-f8f88bb73af1" containerID="4107a32bba11d0de583a4bdfa2178d87f0ed6be0934ddf8b9b8e884e2829a0fe" exitCode=60 Nov 03 23:00:05.899771 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.899732 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" event={"ID":"ba749010-2688-407c-b125-f8f88bb73af1","Type":"ContainerDied","Data":"4107a32bba11d0de583a4bdfa2178d87f0ed6be0934ddf8b9b8e884e2829a0fe"} Nov 03 23:00:05.900005 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.899988 4903 scope.go:122] "RemoveContainer" containerID="4107a32bba11d0de583a4bdfa2178d87f0ed6be0934ddf8b9b8e884e2829a0fe" Nov 03 23:00:05.901720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.901697 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-9b67b1" event={"ID":"eeea961c-52d6-4cf9-87b8-10a7b1dad6b0","Type":"ContainerStarted","Data":"fee6e19d300bf5495a0c0fe6dee645e741ff6ad126c27f077a084b0df341db74"} Nov 03 23:00:05.904346 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.904272 4903 generic.go:356] "Generic (PLEG): container finished" podID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerID="8b1a91f54335b81c13910f6bee7c80d760d9b895b11f1b58d18f3fff7487e7bd" exitCode=60 Nov 03 23:00:05.904446 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.904375 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" event={"ID":"e6f10255-ffed-4e46-b2b9-567ce601cc11","Type":"ContainerDied","Data":"8b1a91f54335b81c13910f6bee7c80d760d9b895b11f1b58d18f3fff7487e7bd"} Nov 03 23:00:05.904649 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.904627 4903 scope.go:122] "RemoveContainer" containerID="8b1a91f54335b81c13910f6bee7c80d760d9b895b11f1b58d18f3fff7487e7bd" Nov 03 23:00:05.907041 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.907017 4903 generic.go:356] "Generic (PLEG): container finished" podID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerID="c2d0dec49b25fd9d9c106df9237bd65ac3a280d2ac7798c32d52070075058ce9" exitCode=60 Nov 03 23:00:05.907146 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.907100 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" event={"ID":"c4c07b85-9bec-46d6-a99c-8a703d94e303","Type":"ContainerDied","Data":"c2d0dec49b25fd9d9c106df9237bd65ac3a280d2ac7798c32d52070075058ce9"} Nov 03 23:00:05.907341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.907321 4903 scope.go:122] "RemoveContainer" containerID="c2d0dec49b25fd9d9c106df9237bd65ac3a280d2ac7798c32d52070075058ce9" Nov 03 23:00:05.909406 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.909379 4903 generic.go:356] "Generic (PLEG): container finished" podID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerID="7db9e6de64351ba54f67b4f313baa153db4f670a4fff5d4eed8b3e3cdb99ddbc" exitCode=60 Nov 03 23:00:05.909504 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.909420 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" event={"ID":"a74ad0b8-94d4-4fbf-9443-d362d91bec33","Type":"ContainerDied","Data":"7db9e6de64351ba54f67b4f313baa153db4f670a4fff5d4eed8b3e3cdb99ddbc"} Nov 03 23:00:05.909664 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.909647 4903 scope.go:122] "RemoveContainer" containerID="7db9e6de64351ba54f67b4f313baa153db4f670a4fff5d4eed8b3e3cdb99ddbc" Nov 03 23:00:05.909919 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:05.909894 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4_odh-ai-gateway-infra(a74ad0b8-94d4-4fbf-9443-d362d91bec33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" Nov 03 23:00:05.918443 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:05.918006 4903 scope.go:122] "RemoveContainer" containerID="e96aa0c5d23c9ac69019da186783f5ee7df6e810353bf45c8d91f9904f0197eb" Nov 03 23:00:06.232612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.232580 4903 scope.go:122] "RemoveContainer" containerID="5f80635314513674d11360bec223be2700be5031cabf7fe0673834e73aad7b48" Nov 03 23:00:06.267554 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.267522 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht"] Nov 03 23:00:06.337667 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.333213 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-9b67b1" podStartSLOduration=2.333193919 podStartE2EDuration="2.333193919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:06.32983899 +0000 UTC m=+2832.733779967" watchObservedRunningTime="2026-11-03 23:00:06.333193919 +0000 UTC m=+2832.737134896" Nov 03 23:00:06.337667 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.335215 4903 scope.go:122] "RemoveContainer" containerID="2092f7e092ac6337a770a2ae2fa4c5bd13bbe76e94987593cd6872b40265e621" Nov 03 23:00:06.525589 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.523889 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8"] Nov 03 23:00:06.583386 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.583359 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5"] Nov 03 23:00:06.583545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.583403 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8"] Nov 03 23:00:06.583545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.583512 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:06.601684 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.601608 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f48f46-dockercfg-5t54j\"" Nov 03 23:00:06.601800 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.601771 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f48f46\"" Nov 03 23:00:06.669988 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.669952 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fdf073-9c1e-4695-8ce6-af0287391ef3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8\" (UID: \"13fdf073-9c1e-4695-8ce6-af0287391ef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:06.670175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.670021 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfxd\" (UniqueName: \"kubernetes.io/projected/13fdf073-9c1e-4695-8ce6-af0287391ef3-kube-api-access-nnfxd\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8\" (UID: \"13fdf073-9c1e-4695-8ce6-af0287391ef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:06.683936 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.683899 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f"] Nov 03 23:00:06.690606 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.690580 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:06.711014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.710984 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f"] Nov 03 23:00:06.787774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.787144 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fdf073-9c1e-4695-8ce6-af0287391ef3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8\" (UID: \"13fdf073-9c1e-4695-8ce6-af0287391ef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:06.787774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.787216 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfxd\" (UniqueName: \"kubernetes.io/projected/13fdf073-9c1e-4695-8ce6-af0287391ef3-kube-api-access-nnfxd\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8\" (UID: \"13fdf073-9c1e-4695-8ce6-af0287391ef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:06.788710 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.788679 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fdf073-9c1e-4695-8ce6-af0287391ef3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8\" (UID: \"13fdf073-9c1e-4695-8ce6-af0287391ef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:06.833087 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.831803 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfxd\" (UniqueName: \"kubernetes.io/projected/13fdf073-9c1e-4695-8ce6-af0287391ef3-kube-api-access-nnfxd\") pod \"payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8\" (UID: \"13fdf073-9c1e-4695-8ce6-af0287391ef3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:06.902439 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.901933 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f\" (UID: \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:06.902439 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.902149 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhnmx\" (UniqueName: \"kubernetes.io/projected/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-kube-api-access-hhnmx\") pod \"payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f\" (UID: \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:06.903037 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.902943 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:06.933335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.929969 4903 scope.go:122] "RemoveContainer" containerID="6d9ebfaa16f6ded18409b177ff30da2e31e56a745b07d4e4cfedeb7a2c9a6ece" Nov 03 23:00:06.933335 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.930404 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v_odh-ai-gateway-infra(b3869c86-0acc-4c84-9438-10f98b688ac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" Nov 03 23:00:06.933335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.933222 4903 generic.go:356] "Generic (PLEG): container finished" podID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerID="7a115f91d62e883791696d4da2e03a7288ab2b88b24a23074defbaf4b180c2e5" exitCode=60 Nov 03 23:00:06.933335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.933326 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" event={"ID":"549b99cf-bb7b-4d6e-b284-060f5a0d144d","Type":"ContainerDied","Data":"7a115f91d62e883791696d4da2e03a7288ab2b88b24a23074defbaf4b180c2e5"} Nov 03 23:00:06.933705 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.933379 4903 scope.go:122] "RemoveContainer" containerID="3b4e824e99c054eb7ba073354ddda76d3f74e1f25a8518bf134bb7261faba4c8" Nov 03 23:00:06.934503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.933815 4903 scope.go:122] "RemoveContainer" containerID="7a115f91d62e883791696d4da2e03a7288ab2b88b24a23074defbaf4b180c2e5" Nov 03 23:00:06.934503 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.934190 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm_odh-ai-gateway-infra(549b99cf-bb7b-4d6e-b284-060f5a0d144d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" Nov 03 23:00:06.953908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.945534 4903 generic.go:356] "Generic (PLEG): container finished" podID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerID="1dad136232d2851b50846fddb733fbfb68fbf153b4ccdc733dd6669695dfc98b" exitCode=60 Nov 03 23:00:06.953908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.945609 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" event={"ID":"bee443a3-007b-46c1-bdea-21ab7ea099db","Type":"ContainerDied","Data":"1dad136232d2851b50846fddb733fbfb68fbf153b4ccdc733dd6669695dfc98b"} Nov 03 23:00:06.953908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.946031 4903 scope.go:122] "RemoveContainer" containerID="1dad136232d2851b50846fddb733fbfb68fbf153b4ccdc733dd6669695dfc98b" Nov 03 23:00:06.953908 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.946610 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89_odh-ai-gateway-infra(bee443a3-007b-46c1-bdea-21ab7ea099db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" Nov 03 23:00:06.953908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.951586 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" event={"ID":"42a165c7-65a1-4e2d-9b8b-10d962afbf1b","Type":"ContainerStarted","Data":"797c2c9bfe43779fa5c8902465d183038f989ea93ccef981736ba86d20468e24"} Nov 03 23:00:06.961213 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.954791 4903 scope.go:122] "RemoveContainer" containerID="3ca9b7952bd2dd749803274d9af2021e7e78b835217d747cc3162bf08021c22f" Nov 03 23:00:06.961213 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.955118 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm_odh-ai-gateway-infra(b76eb079-d83d-4467-873a-a77e020303b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" podUID="b76eb079-d83d-4467-873a-a77e020303b9" Nov 03 23:00:06.961213 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.957741 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" event={"ID":"dc69a099-02c2-45bd-aa5a-516e0680c0d2","Type":"ContainerStarted","Data":"2322976891e16e9715e82df91a63450e0f3d85e4a68f4762806a56bfefcf1256"} Nov 03 23:00:06.961213 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.960976 4903 generic.go:356] "Generic (PLEG): container finished" podID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerID="6f58b70a86683b45936d04ea4b22f2b06b37fe4a956760a0230b17c6c75c48e0" exitCode=60 Nov 03 23:00:06.961213 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.961059 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" event={"ID":"55b2ad97-84d0-4500-8645-172ddb9b1aa4","Type":"ContainerDied","Data":"6f58b70a86683b45936d04ea4b22f2b06b37fe4a956760a0230b17c6c75c48e0"} Nov 03 23:00:06.961617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.961460 4903 scope.go:122] "RemoveContainer" containerID="6f58b70a86683b45936d04ea4b22f2b06b37fe4a956760a0230b17c6c75c48e0" Nov 03 23:00:06.961797 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.961770 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz_odh-ai-gateway-infra(55b2ad97-84d0-4500-8645-172ddb9b1aa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" Nov 03 23:00:06.965815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.965080 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" event={"ID":"ba749010-2688-407c-b125-f8f88bb73af1","Type":"ContainerStarted","Data":"88b61cc5d5c31cb56a2c8542070469f7d9da1860643dac5dc7d1d39114dff896"} Nov 03 23:00:06.968208 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.968149 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-9b67b1" event={"ID":"eeea961c-52d6-4cf9-87b8-10a7b1dad6b0","Type":"ContainerStarted","Data":"d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13"} Nov 03 23:00:06.972857 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.972660 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" event={"ID":"7423e5ac-0e71-463a-b099-f7af3f31a1d8","Type":"ContainerStarted","Data":"184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b"} Nov 03 23:00:06.972857 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.972728 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" event={"ID":"7423e5ac-0e71-463a-b099-f7af3f31a1d8","Type":"ContainerStarted","Data":"d9e8fbad43f848e9901aa2c6ddd8f1e70fd3197c14abaac1007b0f8db004d8c5"} Nov 03 23:00:06.973357 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.973326 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:06.976367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.975715 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" event={"ID":"e6f10255-ffed-4e46-b2b9-567ce601cc11","Type":"ContainerStarted","Data":"c84b421e24fe73f42a10d7143460e977bbd3080023ed9ccfea64a703244ba8b2"} Nov 03 23:00:06.978917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.978413 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" event={"ID":"c4c07b85-9bec-46d6-a99c-8a703d94e303","Type":"ContainerStarted","Data":"1651b0737259c5b9b3a535d862612149e86cca7d71ae23aa837e251f598f90bd"} Nov 03 23:00:06.980076 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.980023 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" event={"ID":"8e2ce3a1-52cd-440b-b120-4d3953b41bdf","Type":"ContainerStarted","Data":"9e62bb12a3e0ec19018257f99a2300013e151829e97e64a4292fee54ebe88cba"} Nov 03 23:00:06.982116 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.982092 4903 generic.go:356] "Generic (PLEG): container finished" podID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerID="ad3c9c6466cca0153627779cf0b49feb323fd0711b80600c5007e00eef4c55df" exitCode=60 Nov 03 23:00:06.982224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.982172 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" event={"ID":"d7f11ee4-b643-4fa5-b040-6e3532ee98a3","Type":"ContainerDied","Data":"ad3c9c6466cca0153627779cf0b49feb323fd0711b80600c5007e00eef4c55df"} Nov 03 23:00:06.982506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.982488 4903 scope.go:122] "RemoveContainer" containerID="ad3c9c6466cca0153627779cf0b49feb323fd0711b80600c5007e00eef4c55df" Nov 03 23:00:06.982754 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.982732 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p_odh-ai-gateway-infra(d7f11ee4-b643-4fa5-b040-6e3532ee98a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" Nov 03 23:00:06.985056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.985033 4903 scope.go:122] "RemoveContainer" containerID="7db9e6de64351ba54f67b4f313baa153db4f670a4fff5d4eed8b3e3cdb99ddbc" Nov 03 23:00:06.985365 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.985345 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4_odh-ai-gateway-infra(a74ad0b8-94d4-4fbf-9443-d362d91bec33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" Nov 03 23:00:06.988377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.988248 4903 generic.go:356] "Generic (PLEG): container finished" podID="933ae1de-a9e1-44da-b47d-9d6177640711" containerID="b124b20c55db1a33b7c9cc58e7f4834a49205e982eeb11c54dcc8f7e877854de" exitCode=60 Nov 03 23:00:06.988377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.988350 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" event={"ID":"933ae1de-a9e1-44da-b47d-9d6177640711","Type":"ContainerDied","Data":"b124b20c55db1a33b7c9cc58e7f4834a49205e982eeb11c54dcc8f7e877854de"} Nov 03 23:00:06.988669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.988648 4903 scope.go:122] "RemoveContainer" containerID="b124b20c55db1a33b7c9cc58e7f4834a49205e982eeb11c54dcc8f7e877854de" Nov 03 23:00:06.988983 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.988930 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz_odh-ai-gateway-infra(933ae1de-a9e1-44da-b47d-9d6177640711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" Nov 03 23:00:06.990995 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.990977 4903 scope.go:122] "RemoveContainer" containerID="7e3eda9b1a703b437c64eb2ce72ec589560bf29b809c43f9d79cd431fa6ee25e" Nov 03 23:00:06.991241 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.991219 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm_odh-ai-gateway-infra(9d9dda56-7458-454f-b28a-7b1122f5deed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" Nov 03 23:00:06.993776 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:06.993420 4903 scope.go:122] "RemoveContainer" containerID="403465c115220ce049b9fb91255e3bd10c57313fe9d9fecd363789737c1abbc8" Nov 03 23:00:06.993776 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:06.993696 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw_odh-ai-gateway-infra(390914dc-6339-4110-a8f0-d3c43479e328)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" podUID="390914dc-6339-4110-a8f0-d3c43479e328" Nov 03 23:00:07.009179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.008206 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f\" (UID: \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:07.010688 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.010655 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f\" (UID: \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:07.011312 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.011268 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnmx\" (UniqueName: \"kubernetes.io/projected/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-kube-api-access-hhnmx\") pod \"payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f\" (UID: \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:07.054069 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.053993 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhnmx\" (UniqueName: \"kubernetes.io/projected/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-kube-api-access-hhnmx\") pod \"payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f\" (UID: \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:07.106196 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.106165 4903 scope.go:122] "RemoveContainer" containerID="492590ade3179f808e0b9eb11b70f23f080f479e97b65ad444f1102e1097dbc4" Nov 03 23:00:07.116483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.115969 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:07.176601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.176503 4903 scope.go:122] "RemoveContainer" containerID="8235dfd97a1877afef764f0fc00a3e700a8d12b83007aa290eb773e89b89919d" Nov 03 23:00:07.228708 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.228673 4903 scope.go:122] "RemoveContainer" containerID="26d99b868aed98b4c6ecca6c6a988e541671f01a0b4261eee680ce98472b27c7" Nov 03 23:00:07.261985 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.261963 4903 scope.go:122] "RemoveContainer" containerID="f62fa6154d9ffe4bb5b6a8963f561aa005cdba61b677aa2870c9a53276de4723" Nov 03 23:00:07.265665 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:07.265635 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fdf073_9c1e_4695_8ce6_af0287391ef3.slice/crio-9954c8debf809286b2fbd800d92430f8a8b601a5ce71f52e86eea2bc6a6393d3 WatchSource:0}: Error finding container 9954c8debf809286b2fbd800d92430f8a8b601a5ce71f52e86eea2bc6a6393d3: Status 404 returned error can't find the container with id 9954c8debf809286b2fbd800d92430f8a8b601a5ce71f52e86eea2bc6a6393d3 Nov 03 23:00:07.269891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.269869 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8"] Nov 03 23:00:07.459520 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.459414 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" podStartSLOduration=2.459284763 podStartE2EDuration="2.459284763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:07.453597004 +0000 UTC m=+2833.857538004" watchObservedRunningTime="2026-11-03 23:00:07.459284763 +0000 UTC m=+2833.863225740" Nov 03 23:00:07.501274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.501240 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-9b67b1"] Nov 03 23:00:07.506061 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.505940 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-9b67b1"] Nov 03 23:00:07.510724 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.510609 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f"] Nov 03 23:00:07.645781 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.645749 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7257-82644f"] Nov 03 23:00:07.646626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.646607 4903 cpu_manager.go:414] "containerMap: removing container" podUID="eeea961c-52d6-4cf9-87b8-10a7b1dad6b0" containerName="test-curl-7257-9b67b1" Nov 03 23:00:07.646626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.646625 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeea961c-52d6-4cf9-87b8-10a7b1dad6b0" containerName="test-curl-7257-9b67b1" Nov 03 23:00:07.646836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.646803 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeea961c-52d6-4cf9-87b8-10a7b1dad6b0" containerName="test-curl-7257-9b67b1" Nov 03 23:00:07.651518 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.651495 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:07.665571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.665544 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-82644f"] Nov 03 23:00:07.734700 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.734653 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sksh6\" (UniqueName: \"kubernetes.io/projected/161f98db-1817-43a5-b52c-5b04e30f303c-kube-api-access-sksh6\") pod \"test-curl-7257-82644f\" (UID: \"161f98db-1817-43a5-b52c-5b04e30f303c\") " pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:07.836513 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.836272 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sksh6\" (UniqueName: \"kubernetes.io/projected/161f98db-1817-43a5-b52c-5b04e30f303c-kube-api-access-sksh6\") pod \"test-curl-7257-82644f\" (UID: \"161f98db-1817-43a5-b52c-5b04e30f303c\") " pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:07.857971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:07.857939 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sksh6\" (UniqueName: \"kubernetes.io/projected/161f98db-1817-43a5-b52c-5b04e30f303c-kube-api-access-sksh6\") pod \"test-curl-7257-82644f\" (UID: \"161f98db-1817-43a5-b52c-5b04e30f303c\") " pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:08.007671 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.007636 4903 generic.go:356] "Generic (PLEG): container finished" podID="ba749010-2688-407c-b125-f8f88bb73af1" containerID="88b61cc5d5c31cb56a2c8542070469f7d9da1860643dac5dc7d1d39114dff896" exitCode=60 Nov 03 23:00:08.008123 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.007719 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" event={"ID":"ba749010-2688-407c-b125-f8f88bb73af1","Type":"ContainerDied","Data":"88b61cc5d5c31cb56a2c8542070469f7d9da1860643dac5dc7d1d39114dff896"} Nov 03 23:00:08.008123 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.007787 4903 scope.go:122] "RemoveContainer" containerID="4107a32bba11d0de583a4bdfa2178d87f0ed6be0934ddf8b9b8e884e2829a0fe" Nov 03 23:00:08.008123 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.008067 4903 scope.go:122] "RemoveContainer" containerID="88b61cc5d5c31cb56a2c8542070469f7d9da1860643dac5dc7d1d39114dff896" Nov 03 23:00:08.008509 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.008487 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt_odh-ai-gateway-infra(ba749010-2688-407c-b125-f8f88bb73af1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" podUID="ba749010-2688-407c-b125-f8f88bb73af1" Nov 03 23:00:08.011456 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.011434 4903 generic.go:356] "Generic (PLEG): container finished" podID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerID="c84b421e24fe73f42a10d7143460e977bbd3080023ed9ccfea64a703244ba8b2" exitCode=60 Nov 03 23:00:08.011579 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.011534 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" event={"ID":"e6f10255-ffed-4e46-b2b9-567ce601cc11","Type":"ContainerDied","Data":"c84b421e24fe73f42a10d7143460e977bbd3080023ed9ccfea64a703244ba8b2"} Nov 03 23:00:08.011794 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.011781 4903 scope.go:122] "RemoveContainer" containerID="c84b421e24fe73f42a10d7143460e977bbd3080023ed9ccfea64a703244ba8b2" Nov 03 23:00:08.012143 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.012123 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb_odh-ai-gateway-infra(e6f10255-ffed-4e46-b2b9-567ce601cc11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" Nov 03 23:00:08.013770 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.013746 4903 generic.go:356] "Generic (PLEG): container finished" podID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerID="1651b0737259c5b9b3a535d862612149e86cca7d71ae23aa837e251f598f90bd" exitCode=60 Nov 03 23:00:08.013880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.013833 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" event={"ID":"c4c07b85-9bec-46d6-a99c-8a703d94e303","Type":"ContainerDied","Data":"1651b0737259c5b9b3a535d862612149e86cca7d71ae23aa837e251f598f90bd"} Nov 03 23:00:08.014087 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.014069 4903 scope.go:122] "RemoveContainer" containerID="1651b0737259c5b9b3a535d862612149e86cca7d71ae23aa837e251f598f90bd" Nov 03 23:00:08.014412 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.014391 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb_odh-ai-gateway-infra(c4c07b85-9bec-46d6-a99c-8a703d94e303)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" Nov 03 23:00:08.016364 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.016319 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" event={"ID":"8e2ce3a1-52cd-440b-b120-4d3953b41bdf","Type":"ContainerStarted","Data":"3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b"} Nov 03 23:00:08.016498 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.016482 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:08.020094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.019770 4903 scope.go:122] "RemoveContainer" containerID="ad3c9c6466cca0153627779cf0b49feb323fd0711b80600c5007e00eef4c55df" Nov 03 23:00:08.020094 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.020039 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p_odh-ai-gateway-infra(d7f11ee4-b643-4fa5-b040-6e3532ee98a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" Nov 03 23:00:08.022563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.022533 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" event={"ID":"b2341ba1-d0ba-460a-ab1e-58a6fd17e217","Type":"ContainerStarted","Data":"63fd0b86e9622ac137e14ec25c3fbef3c4b88c31c37e3017b9aed04db32404aa"} Nov 03 23:00:08.022665 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.022578 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" event={"ID":"b2341ba1-d0ba-460a-ab1e-58a6fd17e217","Type":"ContainerStarted","Data":"738437fe8637fb183ba28f22a00e378fa007403d4521d4e888a59b4825890008"} Nov 03 23:00:08.022734 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.022705 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:08.025405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.025384 4903 scope.go:122] "RemoveContainer" containerID="b124b20c55db1a33b7c9cc58e7f4834a49205e982eeb11c54dcc8f7e877854de" Nov 03 23:00:08.025660 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.025635 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz_odh-ai-gateway-infra(933ae1de-a9e1-44da-b47d-9d6177640711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" Nov 03 23:00:08.027390 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.027201 4903 scope.go:122] "RemoveContainer" containerID="8b1a91f54335b81c13910f6bee7c80d760d9b895b11f1b58d18f3fff7487e7bd" Nov 03 23:00:08.027711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.027691 4903 generic.go:356] "Generic (PLEG): container finished" podID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerID="500a40c05cb1f8e8b322931939ec299e8f97ef1549da88fc9f9720fde1036525" exitCode=60 Nov 03 23:00:08.027935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.027820 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" event={"ID":"08fcf757-4727-4a59-8e4c-b07ad317df69","Type":"ContainerDied","Data":"500a40c05cb1f8e8b322931939ec299e8f97ef1549da88fc9f9720fde1036525"} Nov 03 23:00:08.028060 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.028037 4903 scope.go:122] "RemoveContainer" containerID="500a40c05cb1f8e8b322931939ec299e8f97ef1549da88fc9f9720fde1036525" Nov 03 23:00:08.028678 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.028651 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz_odh-ai-gateway-infra(08fcf757-4727-4a59-8e4c-b07ad317df69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" Nov 03 23:00:08.030819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.030799 4903 scope.go:122] "RemoveContainer" containerID="7a115f91d62e883791696d4da2e03a7288ab2b88b24a23074defbaf4b180c2e5" Nov 03 23:00:08.031032 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.031011 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm_odh-ai-gateway-infra(549b99cf-bb7b-4d6e-b284-060f5a0d144d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" Nov 03 23:00:08.032585 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.032567 4903 scope.go:122] "RemoveContainer" containerID="1dad136232d2851b50846fddb733fbfb68fbf153b4ccdc733dd6669695dfc98b" Nov 03 23:00:08.032931 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.032787 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89_odh-ai-gateway-infra(bee443a3-007b-46c1-bdea-21ab7ea099db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" Nov 03 23:00:08.034483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.034372 4903 generic.go:356] "Generic (PLEG): container finished" podID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerID="797c2c9bfe43779fa5c8902465d183038f989ea93ccef981736ba86d20468e24" exitCode=60 Nov 03 23:00:08.034483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.034395 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" event={"ID":"42a165c7-65a1-4e2d-9b8b-10d962afbf1b","Type":"ContainerDied","Data":"797c2c9bfe43779fa5c8902465d183038f989ea93ccef981736ba86d20468e24"} Nov 03 23:00:08.034483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.034428 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:08.034705 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.034655 4903 scope.go:122] "RemoveContainer" containerID="797c2c9bfe43779fa5c8902465d183038f989ea93ccef981736ba86d20468e24" Nov 03 23:00:08.034885 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.034860 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5_odh-ai-gateway-infra(42a165c7-65a1-4e2d-9b8b-10d962afbf1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" Nov 03 23:00:08.036492 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.036468 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" event={"ID":"13fdf073-9c1e-4695-8ce6-af0287391ef3","Type":"ContainerStarted","Data":"afe482f5b49c809ea69d9b48fbf412ef2f3e5382bcc6aede610399770197da0f"} Nov 03 23:00:08.036618 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.036515 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" event={"ID":"13fdf073-9c1e-4695-8ce6-af0287391ef3","Type":"ContainerStarted","Data":"9954c8debf809286b2fbd800d92430f8a8b601a5ce71f52e86eea2bc6a6393d3"} Nov 03 23:00:08.036618 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.036545 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:08.039178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.039155 4903 generic.go:356] "Generic (PLEG): container finished" podID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerID="2322976891e16e9715e82df91a63450e0f3d85e4a68f4762806a56bfefcf1256" exitCode=60 Nov 03 23:00:08.039632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.039614 4903 scope.go:122] "RemoveContainer" containerID="2322976891e16e9715e82df91a63450e0f3d85e4a68f4762806a56bfefcf1256" Nov 03 23:00:08.039897 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.039870 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr_odh-ai-gateway-infra(dc69a099-02c2-45bd-aa5a-516e0680c0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" Nov 03 23:00:08.039977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.039918 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" event={"ID":"dc69a099-02c2-45bd-aa5a-516e0680c0d2","Type":"ContainerDied","Data":"2322976891e16e9715e82df91a63450e0f3d85e4a68f4762806a56bfefcf1256"} Nov 03 23:00:08.042114 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.041848 4903 scope.go:122] "RemoveContainer" containerID="6f58b70a86683b45936d04ea4b22f2b06b37fe4a956760a0230b17c6c75c48e0" Nov 03 23:00:08.042114 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:08.042078 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz_odh-ai-gateway-infra(55b2ad97-84d0-4500-8645-172ddb9b1aa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" Nov 03 23:00:08.050547 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.050527 4903 scope.go:122] "RemoveContainer" containerID="c2d0dec49b25fd9d9c106df9237bd65ac3a280d2ac7798c32d52070075058ce9" Nov 03 23:00:08.091343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.088799 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" podStartSLOduration=2.088779375 podStartE2EDuration="2.088779375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:08.081799408 +0000 UTC m=+2834.485740387" watchObservedRunningTime="2026-11-03 23:00:08.088779375 +0000 UTC m=+2834.492720357" Nov 03 23:00:08.104442 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.104414 4903 scope.go:122] "RemoveContainer" containerID="4f760807dfb39b3d25a93f0d00c8e527034a89c51772993d8b93154e015b4bd4" Nov 03 23:00:08.136581 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.136471 4903 scope.go:122] "RemoveContainer" containerID="acbf5f9bc9ac06ea8a9b9a24eb2358ea102ddd6c29bbe3dd332b985f7c6a9ec2" Nov 03 23:00:08.170654 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.170628 4903 scope.go:122] "RemoveContainer" containerID="ae5d5d821663e0b1e8c4a922a940445fe97d2ba0a4c66bd27ae2dd00e1068107" Nov 03 23:00:08.224881 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.224828 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" podStartSLOduration=3.224810632 podStartE2EDuration="3.224810632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:08.221992943 +0000 UTC m=+2834.625933918" watchObservedRunningTime="2026-11-03 23:00:08.224810632 +0000 UTC m=+2834.628751607" Nov 03 23:00:08.384341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:08.383180 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7257-82644f"] Nov 03 23:00:08.386781 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:08.386441 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod161f98db_1817_43a5_b52c_5b04e30f303c.slice/crio-1d8ae5b9dbf5afb66dfbb880f4ab9f27685d20bc744ce2397c6042ac559693c1 WatchSource:0}: Error finding container 1d8ae5b9dbf5afb66dfbb880f4ab9f27685d20bc744ce2397c6042ac559693c1: Status 404 returned error can't find the container with id 1d8ae5b9dbf5afb66dfbb880f4ab9f27685d20bc744ce2397c6042ac559693c1 Nov 03 23:00:09.058850 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.058816 4903 scope.go:122] "RemoveContainer" containerID="500a40c05cb1f8e8b322931939ec299e8f97ef1549da88fc9f9720fde1036525" Nov 03 23:00:09.059436 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:09.059162 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz_odh-ai-gateway-infra(08fcf757-4727-4a59-8e4c-b07ad317df69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" Nov 03 23:00:09.061328 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.061274 4903 scope.go:122] "RemoveContainer" containerID="797c2c9bfe43779fa5c8902465d183038f989ea93ccef981736ba86d20468e24" Nov 03 23:00:09.061569 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:09.061546 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5_odh-ai-gateway-infra(42a165c7-65a1-4e2d-9b8b-10d962afbf1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" Nov 03 23:00:09.063141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.063118 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-82644f" event={"ID":"161f98db-1817-43a5-b52c-5b04e30f303c","Type":"ContainerStarted","Data":"ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94"} Nov 03 23:00:09.063235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.063151 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7257-82644f" event={"ID":"161f98db-1817-43a5-b52c-5b04e30f303c","Type":"ContainerStarted","Data":"1d8ae5b9dbf5afb66dfbb880f4ab9f27685d20bc744ce2397c6042ac559693c1"} Nov 03 23:00:09.066037 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.066016 4903 scope.go:122] "RemoveContainer" containerID="2322976891e16e9715e82df91a63450e0f3d85e4a68f4762806a56bfefcf1256" Nov 03 23:00:09.066421 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:09.066378 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr_odh-ai-gateway-infra(dc69a099-02c2-45bd-aa5a-516e0680c0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" Nov 03 23:00:09.068689 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.068672 4903 scope.go:122] "RemoveContainer" containerID="88b61cc5d5c31cb56a2c8542070469f7d9da1860643dac5dc7d1d39114dff896" Nov 03 23:00:09.068900 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:09.068879 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt_odh-ai-gateway-infra(ba749010-2688-407c-b125-f8f88bb73af1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" podUID="ba749010-2688-407c-b125-f8f88bb73af1" Nov 03 23:00:09.070572 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.070555 4903 scope.go:122] "RemoveContainer" containerID="c84b421e24fe73f42a10d7143460e977bbd3080023ed9ccfea64a703244ba8b2" Nov 03 23:00:09.070803 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:09.070783 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb_odh-ai-gateway-infra(e6f10255-ffed-4e46-b2b9-567ce601cc11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" Nov 03 23:00:09.073560 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.073534 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-9b67b1" podUID="eeea961c-52d6-4cf9-87b8-10a7b1dad6b0" containerName="test-curl-7257-9b67b1" containerID="cri-o://d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13" gracePeriod=2 Nov 03 23:00:09.074248 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.073886 4903 scope.go:122] "RemoveContainer" containerID="1651b0737259c5b9b3a535d862612149e86cca7d71ae23aa837e251f598f90bd" Nov 03 23:00:09.074248 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:09.074126 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb_odh-ai-gateway-infra(c4c07b85-9bec-46d6-a99c-8a703d94e303)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" Nov 03 23:00:09.140088 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.140023 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" podStartSLOduration=3.140004101 podStartE2EDuration="3.140004101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:08.563372517 +0000 UTC m=+2834.967313494" watchObservedRunningTime="2026-11-03 23:00:09.140004101 +0000 UTC m=+2835.543945077" Nov 03 23:00:09.287946 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:09.287896 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7257-82644f" podStartSLOduration=2.287881406 podStartE2EDuration="2.287881406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:09.282700691 +0000 UTC m=+2835.686641671" watchObservedRunningTime="2026-11-03 23:00:09.287881406 +0000 UTC m=+2835.691822382" Nov 03 23:00:10.126568 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.126531 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7257-82644f"] Nov 03 23:00:10.131322 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.131275 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7257-82644f"] Nov 03 23:00:10.717696 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.717663 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6"] Nov 03 23:00:10.792472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.718807 4903 cpu_manager.go:414] "containerMap: removing container" podUID="161f98db-1817-43a5-b52c-5b04e30f303c" containerName="test-curl-7257-82644f" Nov 03 23:00:10.792472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.718826 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161f98db-1817-43a5-b52c-5b04e30f303c" containerName="test-curl-7257-82644f" Nov 03 23:00:10.792472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.718981 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="161f98db-1817-43a5-b52c-5b04e30f303c" containerName="test-curl-7257-82644f" Nov 03 23:00:10.792472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.732612 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:10.792472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.755603 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:10.792472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.759407 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6"] Nov 03 23:00:10.792472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.786592 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e268eb-mmbn6\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:10.792472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.786832 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e268eb-mmbn6\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:10.792472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.787124 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swwt7\" (UniqueName: \"kubernetes.io/projected/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-kube-api-access-swwt7\") pod \"maas-api-revoke-sub-73e268eb-mmbn6\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:10.890479 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.889202 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swwt7\" (UniqueName: \"kubernetes.io/projected/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-kube-api-access-swwt7\") pod \"maas-api-revoke-sub-73e268eb-mmbn6\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:10.890479 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.889390 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e268eb-mmbn6\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:10.890479 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.889472 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e268eb-mmbn6\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:10.890479 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.890376 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e268eb-mmbn6\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:10.898124 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.893767 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e268eb-mmbn6\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:10.911268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:10.911237 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swwt7\" (UniqueName: \"kubernetes.io/projected/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-kube-api-access-swwt7\") pod \"maas-api-revoke-sub-73e268eb-mmbn6\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:11.064323 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.064226 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:11.096516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.096483 4903 generic.go:356] "Generic (PLEG): container finished" podID="83503d8f-0a17-4c64-84af-d93426654e30" containerID="b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30" exitCode=0 Nov 03 23:00:11.096675 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.096552 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" event={"ID":"83503d8f-0a17-4c64-84af-d93426654e30","Type":"ContainerDied","Data":"b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30"} Nov 03 23:00:11.096743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.096719 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7257-82644f" podUID="161f98db-1817-43a5-b52c-5b04e30f303c" containerName="test-curl-7257-82644f" containerID="cri-o://ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94" gracePeriod=2 Nov 03 23:00:11.097237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.097218 4903 scope.go:122] "RemoveContainer" containerID="b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30" Nov 03 23:00:11.099256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.099219 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:00:11.099389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.099321 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:00:11.099389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.099376 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:00:11.119911 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:11.116686 4903 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-82644f\" is forbidden: User \"system:node:ip-10-0-133-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-193.ec2.internal' and this object" podUID="161f98db-1817-43a5-b52c-5b04e30f303c" pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:11.176221 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:11.175974 4903 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-82644f\" is forbidden: User \"system:node:ip-10-0-133-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-193.ec2.internal' and this object" podUID="161f98db-1817-43a5-b52c-5b04e30f303c" pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:11.361898 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.361871 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-9b67b1" Nov 03 23:00:11.373799 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:11.373762 4903 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7257-82644f\" is forbidden: User \"system:node:ip-10-0-133-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-193.ec2.internal' and this object" podUID="161f98db-1817-43a5-b52c-5b04e30f303c" pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:11.383240 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.383212 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6"] Nov 03 23:00:11.383617 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:11.383591 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e5ea2b_b9dc_469e_a43b_8e390dd573c2.slice/crio-6b1ce59da2dcdfa4628542f52daf837dca5ce82f12c43306aae5265326b0b2f9 WatchSource:0}: Error finding container 6b1ce59da2dcdfa4628542f52daf837dca5ce82f12c43306aae5265326b0b2f9: Status 404 returned error can't find the container with id 6b1ce59da2dcdfa4628542f52daf837dca5ce82f12c43306aae5265326b0b2f9 Nov 03 23:00:11.503412 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.503370 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeea961c-52d6-4cf9-87b8-10a7b1dad6b0-kube-api-access-qzm4d\" (UniqueName: \"kubernetes.io/projected/eeea961c-52d6-4cf9-87b8-10a7b1dad6b0-kube-api-access-qzm4d\") pod \"eeea961c-52d6-4cf9-87b8-10a7b1dad6b0\" (UID: \"eeea961c-52d6-4cf9-87b8-10a7b1dad6b0\") " Nov 03 23:00:11.505545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.505515 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeea961c-52d6-4cf9-87b8-10a7b1dad6b0-kube-api-access-qzm4d" pod "eeea961c-52d6-4cf9-87b8-10a7b1dad6b0" (UID: "eeea961c-52d6-4cf9-87b8-10a7b1dad6b0"). InnerVolumeSpecName "kube-api-access-qzm4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:11.606423 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.606334 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzm4d\" (UniqueName: \"kubernetes.io/projected/eeea961c-52d6-4cf9-87b8-10a7b1dad6b0-kube-api-access-qzm4d\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:11.739745 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:11.739707 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeea961c-52d6-4cf9-87b8-10a7b1dad6b0" path="/var/lib/kubelet/pods/eeea961c-52d6-4cf9-87b8-10a7b1dad6b0/volumes" Nov 03 23:00:12.106230 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.106193 4903 generic.go:356] "Generic (PLEG): container finished" podID="eeea961c-52d6-4cf9-87b8-10a7b1dad6b0" containerID="d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13" exitCode=137 Nov 03 23:00:12.106454 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.106248 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-9b67b1" Nov 03 23:00:12.106454 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.106342 4903 scope.go:122] "RemoveContainer" containerID="d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13" Nov 03 23:00:12.108287 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.108263 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" event={"ID":"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2","Type":"ContainerStarted","Data":"848945c80b46a9e579c88fdd66bb1ef7b66c0115ae6cb80ae5f3522aa3f017fd"} Nov 03 23:00:12.108506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.108476 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" event={"ID":"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2","Type":"ContainerStarted","Data":"6b1ce59da2dcdfa4628542f52daf837dca5ce82f12c43306aae5265326b0b2f9"} Nov 03 23:00:12.111340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.111320 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" event={"ID":"83503d8f-0a17-4c64-84af-d93426654e30","Type":"ContainerStarted","Data":"4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876"} Nov 03 23:00:12.125823 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.125801 4903 scope.go:122] "RemoveContainer" containerID="d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13" Nov 03 23:00:12.126177 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:12.126148 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13\": container with ID starting with d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13 not found: ID does not exist" containerID="d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13" Nov 03 23:00:12.126321 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.126188 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13"} err="failed to get container status \"d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13\": rpc error: code = NotFound desc = could not find container \"d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13\": container with ID starting with d17b356f61161d5346ef550388231409d78c24958a0ad2dd95f96d9d2c5b5e13 not found: ID does not exist" Nov 03 23:00:12.167590 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.167214 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" podStartSLOduration=2.167192711 podStartE2EDuration="2.167192711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:12.153847137 +0000 UTC m=+2838.557788113" watchObservedRunningTime="2026-11-03 23:00:12.167192711 +0000 UTC m=+2838.571133687" Nov 03 23:00:12.963729 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.963693 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 23:00:12.966326 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.966260 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 23:00:12.966479 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:12.966348 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 23:00:13.126194 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:13.126166 4903 generic.go:356] "Generic (PLEG): container finished" podID="a3e5ea2b-b9dc-469e-a43b-8e390dd573c2" containerID="848945c80b46a9e579c88fdd66bb1ef7b66c0115ae6cb80ae5f3522aa3f017fd" exitCode=0 Nov 03 23:00:13.126356 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:13.126253 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" event={"ID":"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2","Type":"ContainerDied","Data":"848945c80b46a9e579c88fdd66bb1ef7b66c0115ae6cb80ae5f3522aa3f017fd"} Nov 03 23:00:13.405770 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:13.405744 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:13.534013 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:13.533982 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161f98db-1817-43a5-b52c-5b04e30f303c-kube-api-access-sksh6\" (UniqueName: \"kubernetes.io/projected/161f98db-1817-43a5-b52c-5b04e30f303c-kube-api-access-sksh6\") pod \"161f98db-1817-43a5-b52c-5b04e30f303c\" (UID: \"161f98db-1817-43a5-b52c-5b04e30f303c\") " Nov 03 23:00:13.536179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:13.536155 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161f98db-1817-43a5-b52c-5b04e30f303c-kube-api-access-sksh6" pod "161f98db-1817-43a5-b52c-5b04e30f303c" (UID: "161f98db-1817-43a5-b52c-5b04e30f303c"). InnerVolumeSpecName "kube-api-access-sksh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:13.636243 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:13.636209 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sksh6\" (UniqueName: \"kubernetes.io/projected/161f98db-1817-43a5-b52c-5b04e30f303c-kube-api-access-sksh6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:13.773471 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:13.773416 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161f98db-1817-43a5-b52c-5b04e30f303c" path="/var/lib/kubelet/pods/161f98db-1817-43a5-b52c-5b04e30f303c/volumes" Nov 03 23:00:13.966264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:13.966229 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 23:00:13.966716 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:13.966321 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 23:00:14.136646 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.136609 4903 generic.go:356] "Generic (PLEG): container finished" podID="161f98db-1817-43a5-b52c-5b04e30f303c" containerID="ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94" exitCode=137 Nov 03 23:00:14.136843 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.136671 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7257-82644f" Nov 03 23:00:14.136843 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.136713 4903 scope.go:122] "RemoveContainer" containerID="ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94" Nov 03 23:00:14.174093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.173940 4903 scope.go:122] "RemoveContainer" containerID="ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94" Nov 03 23:00:14.174367 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:14.174344 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94\": container with ID starting with ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94 not found: ID does not exist" containerID="ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94" Nov 03 23:00:14.174435 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.174378 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94"} err="failed to get container status \"ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94\": rpc error: code = NotFound desc = could not find container \"ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94\": container with ID starting with ff752d157e70563c71b94bc7aa9b00a03e2adfd0fd3d7c5697229ec353e34c94 not found: ID does not exist" Nov 03 23:00:14.386012 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.385986 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:14.479644 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.479549 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-openshift-ca\") pod \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " Nov 03 23:00:14.479831 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.479810 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-kube-api-access-swwt7\" (UniqueName: \"kubernetes.io/projected/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-kube-api-access-swwt7\") pod \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " Nov 03 23:00:14.479921 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.479851 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-xks-ca\") pod \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\" (UID: \"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2\") " Nov 03 23:00:14.480017 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.479988 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-openshift-ca" pod "a3e5ea2b-b9dc-469e-a43b-8e390dd573c2" (UID: "a3e5ea2b-b9dc-469e-a43b-8e390dd573c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:14.480567 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.480539 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:14.482596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.482569 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-xks-ca" pod "a3e5ea2b-b9dc-469e-a43b-8e390dd573c2" (UID: "a3e5ea2b-b9dc-469e-a43b-8e390dd573c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:14.482855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.482830 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-kube-api-access-swwt7" pod "a3e5ea2b-b9dc-469e-a43b-8e390dd573c2" (UID: "a3e5ea2b-b9dc-469e-a43b-8e390dd573c2"). InnerVolumeSpecName "kube-api-access-swwt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:14.582491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.582456 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swwt7\" (UniqueName: \"kubernetes.io/projected/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-kube-api-access-swwt7\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:14.582491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.582489 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:14.960317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.959713 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 23:00:14.960317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:14.959774 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 23:00:15.171196 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:15.171164 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" Nov 03 23:00:15.171711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:15.171200 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6" event={"ID":"a3e5ea2b-b9dc-469e-a43b-8e390dd573c2","Type":"ContainerDied","Data":"6b1ce59da2dcdfa4628542f52daf837dca5ce82f12c43306aae5265326b0b2f9"} Nov 03 23:00:15.171711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:15.171264 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b1ce59da2dcdfa4628542f52daf837dca5ce82f12c43306aae5265326b0b2f9" Nov 03 23:00:15.966781 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:15.966740 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 23:00:15.967009 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:15.966810 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 23:00:16.271978 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:16.271893 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 23:00:16.726076 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:16.726045 4903 scope.go:122] "RemoveContainer" containerID="7e3eda9b1a703b437c64eb2ce72ec589560bf29b809c43f9d79cd431fa6ee25e" Nov 03 23:00:16.726267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:16.726233 4903 scope.go:122] "RemoveContainer" containerID="7db9e6de64351ba54f67b4f313baa153db4f670a4fff5d4eed8b3e3cdb99ddbc" Nov 03 23:00:16.963872 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:16.963839 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Nov 03 23:00:16.964185 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:16.964144 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Nov 03 23:00:17.725776 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:17.725742 4903 scope.go:122] "RemoveContainer" containerID="b124b20c55db1a33b7c9cc58e7f4834a49205e982eeb11c54dcc8f7e877854de" Nov 03 23:00:17.962518 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:17.962493 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 23:00:17.963680 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:17.963658 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 23:00:18.211238 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.211193 4903 generic.go:356] "Generic (PLEG): container finished" podID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerID="57db7e66d9f1c7ff78cbb49c9363d474f9049157f83e743085659f2108d07eed" exitCode=60 Nov 03 23:00:18.211478 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.211265 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" event={"ID":"a74ad0b8-94d4-4fbf-9443-d362d91bec33","Type":"ContainerDied","Data":"57db7e66d9f1c7ff78cbb49c9363d474f9049157f83e743085659f2108d07eed"} Nov 03 23:00:18.211478 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.211375 4903 scope.go:122] "RemoveContainer" containerID="7db9e6de64351ba54f67b4f313baa153db4f670a4fff5d4eed8b3e3cdb99ddbc" Nov 03 23:00:18.211907 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.211862 4903 scope.go:122] "RemoveContainer" containerID="57db7e66d9f1c7ff78cbb49c9363d474f9049157f83e743085659f2108d07eed" Nov 03 23:00:18.212437 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:18.212410 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4_odh-ai-gateway-infra(a74ad0b8-94d4-4fbf-9443-d362d91bec33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" Nov 03 23:00:18.215109 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.215068 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" event={"ID":"933ae1de-a9e1-44da-b47d-9d6177640711","Type":"ContainerStarted","Data":"e0a98a8dd64832d909217f06e8fb769dd7b89ab4bb9c52f7de5e62e79903c9d9"} Nov 03 23:00:18.221471 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.221447 4903 generic.go:356] "Generic (PLEG): container finished" podID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerID="de5006ded376f5e75ac0ed8319b911a208e0e78de38c991db6a7fe4c78f7103e" exitCode=60 Nov 03 23:00:18.221618 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.221538 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" event={"ID":"9d9dda56-7458-454f-b28a-7b1122f5deed","Type":"ContainerDied","Data":"de5006ded376f5e75ac0ed8319b911a208e0e78de38c991db6a7fe4c78f7103e"} Nov 03 23:00:18.222241 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.222219 4903 scope.go:122] "RemoveContainer" containerID="de5006ded376f5e75ac0ed8319b911a208e0e78de38c991db6a7fe4c78f7103e" Nov 03 23:00:18.222615 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:18.222594 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm_odh-ai-gateway-infra(9d9dda56-7458-454f-b28a-7b1122f5deed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" Nov 03 23:00:18.230265 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.230241 4903 scope.go:122] "RemoveContainer" containerID="7e3eda9b1a703b437c64eb2ce72ec589560bf29b809c43f9d79cd431fa6ee25e" Nov 03 23:00:18.386381 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:18.386285 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:00:18.726252 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.726221 4903 scope.go:122] "RemoveContainer" containerID="2322976891e16e9715e82df91a63450e0f3d85e4a68f4762806a56bfefcf1256" Nov 03 23:00:18.726735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.726411 4903 scope.go:122] "RemoveContainer" containerID="3ca9b7952bd2dd749803274d9af2021e7e78b835217d747cc3162bf08021c22f" Nov 03 23:00:18.726735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.726522 4903 scope.go:122] "RemoveContainer" containerID="7a115f91d62e883791696d4da2e03a7288ab2b88b24a23074defbaf4b180c2e5" Nov 03 23:00:18.726735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.726650 4903 scope.go:122] "RemoveContainer" containerID="6d9ebfaa16f6ded18409b177ff30da2e31e56a745b07d4e4cfedeb7a2c9a6ece" Nov 03 23:00:18.726913 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:18.726824 4903 scope.go:122] "RemoveContainer" containerID="6f58b70a86683b45936d04ea4b22f2b06b37fe4a956760a0230b17c6c75c48e0" Nov 03 23:00:19.075512 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.075482 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:00:19.076014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.075990 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:00:19.076517 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.076496 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:00:19.076781 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.076764 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:00:19.234983 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.234843 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" event={"ID":"b3869c86-0acc-4c84-9438-10f98b688ac8","Type":"ContainerStarted","Data":"a42f6ad330aec73374e8b98a8fb58d4b4da05a3d5de8e824b17ec13b8840f210"} Nov 03 23:00:19.238232 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.238202 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" event={"ID":"dc69a099-02c2-45bd-aa5a-516e0680c0d2","Type":"ContainerStarted","Data":"40fe950fbed583b652691ce81a9de0e07c326cc6297ff906441d6523571833fd"} Nov 03 23:00:19.249264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.240983 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" event={"ID":"55b2ad97-84d0-4500-8645-172ddb9b1aa4","Type":"ContainerStarted","Data":"59aa3bc0d669b4a412cd6330f99e6df1e33c75efda4ad158e9070493ab134509"} Nov 03 23:00:19.249264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.246224 4903 generic.go:356] "Generic (PLEG): container finished" podID="933ae1de-a9e1-44da-b47d-9d6177640711" containerID="e0a98a8dd64832d909217f06e8fb769dd7b89ab4bb9c52f7de5e62e79903c9d9" exitCode=60 Nov 03 23:00:19.249264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.246331 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" event={"ID":"933ae1de-a9e1-44da-b47d-9d6177640711","Type":"ContainerDied","Data":"e0a98a8dd64832d909217f06e8fb769dd7b89ab4bb9c52f7de5e62e79903c9d9"} Nov 03 23:00:19.249264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.246392 4903 scope.go:122] "RemoveContainer" containerID="b124b20c55db1a33b7c9cc58e7f4834a49205e982eeb11c54dcc8f7e877854de" Nov 03 23:00:19.249264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.246711 4903 scope.go:122] "RemoveContainer" containerID="e0a98a8dd64832d909217f06e8fb769dd7b89ab4bb9c52f7de5e62e79903c9d9" Nov 03 23:00:19.249264 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:19.247001 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz_odh-ai-gateway-infra(933ae1de-a9e1-44da-b47d-9d6177640711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" Nov 03 23:00:19.307438 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.307403 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4"] Nov 03 23:00:19.357799 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.357753 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" podStartSLOduration=19.357740141 podStartE2EDuration="19.357740141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:19.351357793 +0000 UTC m=+2845.755298769" watchObservedRunningTime="2026-11-03 23:00:19.357740141 +0000 UTC m=+2845.761681116" Nov 03 23:00:19.377640 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.377611 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz"] Nov 03 23:00:19.424315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.423775 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm"] Nov 03 23:00:19.516411 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.516338 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:19.570704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.570666 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74ad0b8-94d4-4fbf-9443-d362d91bec33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74ad0b8-94d4-4fbf-9443-d362d91bec33-maas-api-service-ca\") pod \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\" (UID: \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\") " Nov 03 23:00:19.570853 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.570760 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74ad0b8-94d4-4fbf-9443-d362d91bec33-kube-api-access-2dnfd\" (UniqueName: \"kubernetes.io/projected/a74ad0b8-94d4-4fbf-9443-d362d91bec33-kube-api-access-2dnfd\") pod \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\" (UID: \"a74ad0b8-94d4-4fbf-9443-d362d91bec33\") " Nov 03 23:00:19.573725 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.573689 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74ad0b8-94d4-4fbf-9443-d362d91bec33-maas-api-service-ca" pod "a74ad0b8-94d4-4fbf-9443-d362d91bec33" (UID: "a74ad0b8-94d4-4fbf-9443-d362d91bec33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:19.574412 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.574378 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74ad0b8-94d4-4fbf-9443-d362d91bec33-kube-api-access-2dnfd" pod "a74ad0b8-94d4-4fbf-9443-d362d91bec33" (UID: "a74ad0b8-94d4-4fbf-9443-d362d91bec33"). InnerVolumeSpecName "kube-api-access-2dnfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:19.639712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.639686 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:19.672853 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.672825 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9dda56-7458-454f-b28a-7b1122f5deed-kube-api-access-qn4hr\" (UniqueName: \"kubernetes.io/projected/9d9dda56-7458-454f-b28a-7b1122f5deed-kube-api-access-qn4hr\") pod \"9d9dda56-7458-454f-b28a-7b1122f5deed\" (UID: \"9d9dda56-7458-454f-b28a-7b1122f5deed\") " Nov 03 23:00:19.673004 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.672987 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9dda56-7458-454f-b28a-7b1122f5deed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9dda56-7458-454f-b28a-7b1122f5deed-maas-api-service-ca\") pod \"9d9dda56-7458-454f-b28a-7b1122f5deed\" (UID: \"9d9dda56-7458-454f-b28a-7b1122f5deed\") " Nov 03 23:00:19.673394 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.673367 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9dda56-7458-454f-b28a-7b1122f5deed-maas-api-service-ca" pod "9d9dda56-7458-454f-b28a-7b1122f5deed" (UID: "9d9dda56-7458-454f-b28a-7b1122f5deed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:19.673641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.673628 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9dda56-7458-454f-b28a-7b1122f5deed-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:19.673690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.673647 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74ad0b8-94d4-4fbf-9443-d362d91bec33-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:19.673690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.673656 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnfd\" (UniqueName: \"kubernetes.io/projected/a74ad0b8-94d4-4fbf-9443-d362d91bec33-kube-api-access-2dnfd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:19.674841 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.674812 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9dda56-7458-454f-b28a-7b1122f5deed-kube-api-access-qn4hr" pod "9d9dda56-7458-454f-b28a-7b1122f5deed" (UID: "9d9dda56-7458-454f-b28a-7b1122f5deed"). InnerVolumeSpecName "kube-api-access-qn4hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:19.726601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.726558 4903 scope.go:122] "RemoveContainer" containerID="403465c115220ce049b9fb91255e3bd10c57313fe9d9fecd363789737c1abbc8" Nov 03 23:00:19.774787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:19.774668 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4hr\" (UniqueName: \"kubernetes.io/projected/9d9dda56-7458-454f-b28a-7b1122f5deed-kube-api-access-qn4hr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:20.259546 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.259508 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" event={"ID":"a74ad0b8-94d4-4fbf-9443-d362d91bec33","Type":"ContainerDied","Data":"8b0adadbc28741ae751ec2f345552f6625ff352018693cdffc7b37c047537b18"} Nov 03 23:00:20.259743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.259521 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4" Nov 03 23:00:20.259743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.259610 4903 scope.go:122] "RemoveContainer" containerID="57db7e66d9f1c7ff78cbb49c9363d474f9049157f83e743085659f2108d07eed" Nov 03 23:00:20.263409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.263390 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" Nov 03 23:00:20.263535 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.263393 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm" event={"ID":"9d9dda56-7458-454f-b28a-7b1122f5deed","Type":"ContainerDied","Data":"3f9390ce4524fbae91bc3f9289aa14e65d56188d82e45b5c8b6d632f32b39cc1"} Nov 03 23:00:20.265857 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.265834 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" event={"ID":"390914dc-6339-4110-a8f0-d3c43479e328","Type":"ContainerStarted","Data":"d2cbbb522d396d811d4e74e9a0b58e63064cd8728902441516b00ca02abf5bc0"} Nov 03 23:00:20.267877 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.267852 4903 generic.go:356] "Generic (PLEG): container finished" podID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerID="a42f6ad330aec73374e8b98a8fb58d4b4da05a3d5de8e824b17ec13b8840f210" exitCode=60 Nov 03 23:00:20.267974 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.267914 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" event={"ID":"b3869c86-0acc-4c84-9438-10f98b688ac8","Type":"ContainerDied","Data":"a42f6ad330aec73374e8b98a8fb58d4b4da05a3d5de8e824b17ec13b8840f210"} Nov 03 23:00:20.268269 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.268250 4903 scope.go:122] "RemoveContainer" containerID="a42f6ad330aec73374e8b98a8fb58d4b4da05a3d5de8e824b17ec13b8840f210" Nov 03 23:00:20.268598 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:20.268575 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v_odh-ai-gateway-infra(b3869c86-0acc-4c84-9438-10f98b688ac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" Nov 03 23:00:20.270208 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.270184 4903 generic.go:356] "Generic (PLEG): container finished" podID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerID="50ebdea83cd8e4d147b1b87d50b5c23a40aec62f6aa93faab1c9136d21964f3b" exitCode=60 Nov 03 23:00:20.270334 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.270253 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" event={"ID":"549b99cf-bb7b-4d6e-b284-060f5a0d144d","Type":"ContainerDied","Data":"50ebdea83cd8e4d147b1b87d50b5c23a40aec62f6aa93faab1c9136d21964f3b"} Nov 03 23:00:20.270740 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.270722 4903 scope.go:122] "RemoveContainer" containerID="50ebdea83cd8e4d147b1b87d50b5c23a40aec62f6aa93faab1c9136d21964f3b" Nov 03 23:00:20.271066 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:20.271033 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm_odh-ai-gateway-infra(549b99cf-bb7b-4d6e-b284-060f5a0d144d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" Nov 03 23:00:20.272563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.272536 4903 generic.go:356] "Generic (PLEG): container finished" podID="b76eb079-d83d-4467-873a-a77e020303b9" containerID="bf749a2296c0074adf501f7bbd5aed747a023b200101b434aed768c2a5ade001" exitCode=60 Nov 03 23:00:20.272665 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.272608 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" event={"ID":"b76eb079-d83d-4467-873a-a77e020303b9","Type":"ContainerDied","Data":"bf749a2296c0074adf501f7bbd5aed747a023b200101b434aed768c2a5ade001"} Nov 03 23:00:20.273044 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.273014 4903 scope.go:122] "RemoveContainer" containerID="bf749a2296c0074adf501f7bbd5aed747a023b200101b434aed768c2a5ade001" Nov 03 23:00:20.273394 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:20.273366 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm_odh-ai-gateway-infra(b76eb079-d83d-4467-873a-a77e020303b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" podUID="b76eb079-d83d-4467-873a-a77e020303b9" Nov 03 23:00:20.274478 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.274449 4903 scope.go:122] "RemoveContainer" containerID="de5006ded376f5e75ac0ed8319b911a208e0e78de38c991db6a7fe4c78f7103e" Nov 03 23:00:20.274979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.274959 4903 generic.go:356] "Generic (PLEG): container finished" podID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerID="40fe950fbed583b652691ce81a9de0e07c326cc6297ff906441d6523571833fd" exitCode=60 Nov 03 23:00:20.275072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.274982 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" event={"ID":"dc69a099-02c2-45bd-aa5a-516e0680c0d2","Type":"ContainerDied","Data":"40fe950fbed583b652691ce81a9de0e07c326cc6297ff906441d6523571833fd"} Nov 03 23:00:20.275482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.275458 4903 scope.go:122] "RemoveContainer" containerID="40fe950fbed583b652691ce81a9de0e07c326cc6297ff906441d6523571833fd" Nov 03 23:00:20.275779 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:20.275754 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr_odh-ai-gateway-infra(dc69a099-02c2-45bd-aa5a-516e0680c0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" Nov 03 23:00:20.277881 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.277849 4903 generic.go:356] "Generic (PLEG): container finished" podID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerID="59aa3bc0d669b4a412cd6330f99e6df1e33c75efda4ad158e9070493ab134509" exitCode=60 Nov 03 23:00:20.277982 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.277969 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" event={"ID":"55b2ad97-84d0-4500-8645-172ddb9b1aa4","Type":"ContainerDied","Data":"59aa3bc0d669b4a412cd6330f99e6df1e33c75efda4ad158e9070493ab134509"} Nov 03 23:00:20.278317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.278288 4903 scope.go:122] "RemoveContainer" containerID="59aa3bc0d669b4a412cd6330f99e6df1e33c75efda4ad158e9070493ab134509" Nov 03 23:00:20.278576 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:20.278558 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz_odh-ai-gateway-infra(55b2ad97-84d0-4500-8645-172ddb9b1aa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" Nov 03 23:00:20.289668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.289639 4903 scope.go:122] "RemoveContainer" containerID="6d9ebfaa16f6ded18409b177ff30da2e31e56a745b07d4e4cfedeb7a2c9a6ece" Nov 03 23:00:20.292638 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.292607 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v"] Nov 03 23:00:20.307070 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.306595 4903 scope.go:122] "RemoveContainer" containerID="7a115f91d62e883791696d4da2e03a7288ab2b88b24a23074defbaf4b180c2e5" Nov 03 23:00:20.323604 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.323577 4903 scope.go:122] "RemoveContainer" containerID="3ca9b7952bd2dd749803274d9af2021e7e78b835217d747cc3162bf08021c22f" Nov 03 23:00:20.340571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.340447 4903 scope.go:122] "RemoveContainer" containerID="2322976891e16e9715e82df91a63450e0f3d85e4a68f4762806a56bfefcf1256" Nov 03 23:00:20.366839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.366801 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz"] Nov 03 23:00:20.372157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.372138 4903 scope.go:122] "RemoveContainer" containerID="6f58b70a86683b45936d04ea4b22f2b06b37fe4a956760a0230b17c6c75c48e0" Nov 03 23:00:20.391323 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.391254 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" podStartSLOduration=20.391239892 podStartE2EDuration="20.391239892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:20.386114037 +0000 UTC m=+2846.790055010" watchObservedRunningTime="2026-11-03 23:00:20.391239892 +0000 UTC m=+2846.795180884" Nov 03 23:00:20.403937 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.403909 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr"] Nov 03 23:00:20.424479 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.424455 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:20.462021 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.461990 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4"] Nov 03 23:00:20.467092 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.467064 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-281732-29895780-jt6p4"] Nov 03 23:00:20.484425 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.484392 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933ae1de-a9e1-44da-b47d-9d6177640711-kube-api-access-rv55g\" (UniqueName: \"kubernetes.io/projected/933ae1de-a9e1-44da-b47d-9d6177640711-kube-api-access-rv55g\") pod \"933ae1de-a9e1-44da-b47d-9d6177640711\" (UID: \"933ae1de-a9e1-44da-b47d-9d6177640711\") " Nov 03 23:00:20.484608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.484579 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933ae1de-a9e1-44da-b47d-9d6177640711-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ae1de-a9e1-44da-b47d-9d6177640711-maas-api-service-ca\") pod \"933ae1de-a9e1-44da-b47d-9d6177640711\" (UID: \"933ae1de-a9e1-44da-b47d-9d6177640711\") " Nov 03 23:00:20.484949 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.484921 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933ae1de-a9e1-44da-b47d-9d6177640711-maas-api-service-ca" pod "933ae1de-a9e1-44da-b47d-9d6177640711" (UID: "933ae1de-a9e1-44da-b47d-9d6177640711"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:20.485373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.485353 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/933ae1de-a9e1-44da-b47d-9d6177640711-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:20.486806 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.486627 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933ae1de-a9e1-44da-b47d-9d6177640711-kube-api-access-rv55g" pod "933ae1de-a9e1-44da-b47d-9d6177640711" (UID: "933ae1de-a9e1-44da-b47d-9d6177640711"). InnerVolumeSpecName "kube-api-access-rv55g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:20.532396 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.532363 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm"] Nov 03 23:00:20.537096 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.537069 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-04988c-29895780-86vxm"] Nov 03 23:00:20.586757 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.586680 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv55g\" (UniqueName: \"kubernetes.io/projected/933ae1de-a9e1-44da-b47d-9d6177640711-kube-api-access-rv55g\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:20.726748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.726716 4903 scope.go:122] "RemoveContainer" containerID="ad3c9c6466cca0153627779cf0b49feb323fd0711b80600c5007e00eef4c55df" Nov 03 23:00:20.727179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:20.726805 4903 scope.go:122] "RemoveContainer" containerID="500a40c05cb1f8e8b322931939ec299e8f97ef1549da88fc9f9720fde1036525" Nov 03 23:00:21.297761 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.297723 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" event={"ID":"d7f11ee4-b643-4fa5-b040-6e3532ee98a3","Type":"ContainerStarted","Data":"840c864af532c3472828bc1566e027330e6ad1b72b0df797550dab4479de10ad"} Nov 03 23:00:21.300275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.300250 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" Nov 03 23:00:21.300275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.300262 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz" event={"ID":"933ae1de-a9e1-44da-b47d-9d6177640711","Type":"ContainerDied","Data":"3c02ed3bbdf0385936a908d7d1f509037e14d3e2e01442e134bfb97d18938769"} Nov 03 23:00:21.300479 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.300343 4903 scope.go:122] "RemoveContainer" containerID="e0a98a8dd64832d909217f06e8fb769dd7b89ab4bb9c52f7de5e62e79903c9d9" Nov 03 23:00:21.304248 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.304221 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" event={"ID":"390914dc-6339-4110-a8f0-d3c43479e328","Type":"ContainerDied","Data":"d2cbbb522d396d811d4e74e9a0b58e63064cd8728902441516b00ca02abf5bc0"} Nov 03 23:00:21.304759 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.304736 4903 scope.go:122] "RemoveContainer" containerID="d2cbbb522d396d811d4e74e9a0b58e63064cd8728902441516b00ca02abf5bc0" Nov 03 23:00:21.305091 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:21.305044 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw_odh-ai-gateway-infra(390914dc-6339-4110-a8f0-d3c43479e328)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" podUID="390914dc-6339-4110-a8f0-d3c43479e328" Nov 03 23:00:21.305432 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.305405 4903 generic.go:356] "Generic (PLEG): container finished" podID="390914dc-6339-4110-a8f0-d3c43479e328" containerID="d2cbbb522d396d811d4e74e9a0b58e63064cd8728902441516b00ca02abf5bc0" exitCode=60 Nov 03 23:00:21.343574 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.342428 4903 scope.go:122] "RemoveContainer" containerID="403465c115220ce049b9fb91255e3bd10c57313fe9d9fecd363789737c1abbc8" Nov 03 23:00:21.343574 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.342658 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" event={"ID":"08fcf757-4727-4a59-8e4c-b07ad317df69","Type":"ContainerStarted","Data":"09ecb303ffa3ee300337c40f9e002aeb82c7888101395184e1af84e26b7154f7"} Nov 03 23:00:21.407321 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.406472 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw"] Nov 03 23:00:21.450793 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.450762 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm"] Nov 03 23:00:21.528276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.528239 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:21.530867 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.530839 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm"] Nov 03 23:00:21.553712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.535991 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" podStartSLOduration=21.535973815 podStartE2EDuration="21.535973815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:21.518773416 +0000 UTC m=+2847.922714397" watchObservedRunningTime="2026-11-03 23:00:21.535973815 +0000 UTC m=+2847.939914793" Nov 03 23:00:21.580373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.580339 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz"] Nov 03 23:00:21.590938 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.590886 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-2a2eca-29895780-84fcz"] Nov 03 23:00:21.611449 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.611409 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc69a099-02c2-45bd-aa5a-516e0680c0d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc69a099-02c2-45bd-aa5a-516e0680c0d2-maas-api-service-ca\") pod \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\" (UID: \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\") " Nov 03 23:00:21.611642 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.611562 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc69a099-02c2-45bd-aa5a-516e0680c0d2-kube-api-access-hrs8n\" (UniqueName: \"kubernetes.io/projected/dc69a099-02c2-45bd-aa5a-516e0680c0d2-kube-api-access-hrs8n\") pod \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\" (UID: \"dc69a099-02c2-45bd-aa5a-516e0680c0d2\") " Nov 03 23:00:21.612082 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.612056 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc69a099-02c2-45bd-aa5a-516e0680c0d2-maas-api-service-ca" pod "dc69a099-02c2-45bd-aa5a-516e0680c0d2" (UID: "dc69a099-02c2-45bd-aa5a-516e0680c0d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:21.627500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.627468 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc69a099-02c2-45bd-aa5a-516e0680c0d2-kube-api-access-hrs8n" pod "dc69a099-02c2-45bd-aa5a-516e0680c0d2" (UID: "dc69a099-02c2-45bd-aa5a-516e0680c0d2"). InnerVolumeSpecName "kube-api-access-hrs8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:21.697389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.693876 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:21.699270 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.697880 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:21.713173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.713127 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b2ad97-84d0-4500-8645-172ddb9b1aa4-kube-api-access-8k6s4\" (UniqueName: \"kubernetes.io/projected/55b2ad97-84d0-4500-8645-172ddb9b1aa4-kube-api-access-8k6s4\") pod \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\" (UID: \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\") " Nov 03 23:00:21.713335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.713278 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3869c86-0acc-4c84-9438-10f98b688ac8-kube-api-access-tbts8\" (UniqueName: \"kubernetes.io/projected/b3869c86-0acc-4c84-9438-10f98b688ac8-kube-api-access-tbts8\") pod \"b3869c86-0acc-4c84-9438-10f98b688ac8\" (UID: \"b3869c86-0acc-4c84-9438-10f98b688ac8\") " Nov 03 23:00:21.714077 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.713459 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b2ad97-84d0-4500-8645-172ddb9b1aa4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b2ad97-84d0-4500-8645-172ddb9b1aa4-maas-api-service-ca\") pod \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\" (UID: \"55b2ad97-84d0-4500-8645-172ddb9b1aa4\") " Nov 03 23:00:21.714077 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.713512 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3869c86-0acc-4c84-9438-10f98b688ac8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3869c86-0acc-4c84-9438-10f98b688ac8-maas-api-service-ca\") pod \"b3869c86-0acc-4c84-9438-10f98b688ac8\" (UID: \"b3869c86-0acc-4c84-9438-10f98b688ac8\") " Nov 03 23:00:21.714077 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.713883 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrs8n\" (UniqueName: \"kubernetes.io/projected/dc69a099-02c2-45bd-aa5a-516e0680c0d2-kube-api-access-hrs8n\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:21.714077 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.713897 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc69a099-02c2-45bd-aa5a-516e0680c0d2-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:21.714772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.714746 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b2ad97-84d0-4500-8645-172ddb9b1aa4-maas-api-service-ca" pod "55b2ad97-84d0-4500-8645-172ddb9b1aa4" (UID: "55b2ad97-84d0-4500-8645-172ddb9b1aa4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:21.714904 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.714756 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3869c86-0acc-4c84-9438-10f98b688ac8-maas-api-service-ca" pod "b3869c86-0acc-4c84-9438-10f98b688ac8" (UID: "b3869c86-0acc-4c84-9438-10f98b688ac8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:21.729393 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.729278 4903 scope.go:122] "RemoveContainer" containerID="1dad136232d2851b50846fddb733fbfb68fbf153b4ccdc733dd6669695dfc98b" Nov 03 23:00:21.734667 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.734643 4903 scope.go:122] "RemoveContainer" containerID="88b61cc5d5c31cb56a2c8542070469f7d9da1860643dac5dc7d1d39114dff896" Nov 03 23:00:21.735916 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.735897 4903 scope.go:122] "RemoveContainer" containerID="c84b421e24fe73f42a10d7143460e977bbd3080023ed9ccfea64a703244ba8b2" Nov 03 23:00:21.736709 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.736692 4903 scope.go:122] "RemoveContainer" containerID="797c2c9bfe43779fa5c8902465d183038f989ea93ccef981736ba86d20468e24" Nov 03 23:00:21.740612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.738490 4903 scope.go:122] "RemoveContainer" containerID="1651b0737259c5b9b3a535d862612149e86cca7d71ae23aa837e251f598f90bd" Nov 03 23:00:21.759078 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.758167 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b2ad97-84d0-4500-8645-172ddb9b1aa4-kube-api-access-8k6s4" pod "55b2ad97-84d0-4500-8645-172ddb9b1aa4" (UID: "55b2ad97-84d0-4500-8645-172ddb9b1aa4"). InnerVolumeSpecName "kube-api-access-8k6s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:21.759078 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.758214 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3869c86-0acc-4c84-9438-10f98b688ac8-kube-api-access-tbts8" pod "b3869c86-0acc-4c84-9438-10f98b688ac8" (UID: "b3869c86-0acc-4c84-9438-10f98b688ac8"). InnerVolumeSpecName "kube-api-access-tbts8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:21.769123 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.769097 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" path="/var/lib/kubelet/pods/933ae1de-a9e1-44da-b47d-9d6177640711/volumes" Nov 03 23:00:21.769893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.769872 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" path="/var/lib/kubelet/pods/9d9dda56-7458-454f-b28a-7b1122f5deed/volumes" Nov 03 23:00:21.770561 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.770544 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" path="/var/lib/kubelet/pods/a74ad0b8-94d4-4fbf-9443-d362d91bec33/volumes" Nov 03 23:00:21.817620 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.815363 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k6s4\" (UniqueName: \"kubernetes.io/projected/55b2ad97-84d0-4500-8645-172ddb9b1aa4-kube-api-access-8k6s4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:21.817620 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.815388 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbts8\" (UniqueName: \"kubernetes.io/projected/b3869c86-0acc-4c84-9438-10f98b688ac8-kube-api-access-tbts8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:21.817620 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.815399 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b2ad97-84d0-4500-8645-172ddb9b1aa4-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:21.817620 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.815408 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3869c86-0acc-4c84-9438-10f98b688ac8-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:21.879046 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.879025 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:21.889999 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.889976 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:21.916439 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.916415 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76eb079-d83d-4467-873a-a77e020303b9-kube-api-access-77845\" (UniqueName: \"kubernetes.io/projected/b76eb079-d83d-4467-873a-a77e020303b9-kube-api-access-77845\") pod \"b76eb079-d83d-4467-873a-a77e020303b9\" (UID: \"b76eb079-d83d-4467-873a-a77e020303b9\") " Nov 03 23:00:21.916565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.916482 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549b99cf-bb7b-4d6e-b284-060f5a0d144d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/549b99cf-bb7b-4d6e-b284-060f5a0d144d-maas-api-service-ca\") pod \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\" (UID: \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\") " Nov 03 23:00:21.916565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.916547 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549b99cf-bb7b-4d6e-b284-060f5a0d144d-kube-api-access-ctfb6\" (UniqueName: \"kubernetes.io/projected/549b99cf-bb7b-4d6e-b284-060f5a0d144d-kube-api-access-ctfb6\") pod \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\" (UID: \"549b99cf-bb7b-4d6e-b284-060f5a0d144d\") " Nov 03 23:00:21.917042 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.916727 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76eb079-d83d-4467-873a-a77e020303b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76eb079-d83d-4467-873a-a77e020303b9-maas-api-service-ca\") pod \"b76eb079-d83d-4467-873a-a77e020303b9\" (UID: \"b76eb079-d83d-4467-873a-a77e020303b9\") " Nov 03 23:00:21.917042 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.917003 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549b99cf-bb7b-4d6e-b284-060f5a0d144d-maas-api-service-ca" pod "549b99cf-bb7b-4d6e-b284-060f5a0d144d" (UID: "549b99cf-bb7b-4d6e-b284-060f5a0d144d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:21.917228 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.917125 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76eb079-d83d-4467-873a-a77e020303b9-maas-api-service-ca" pod "b76eb079-d83d-4467-873a-a77e020303b9" (UID: "b76eb079-d83d-4467-873a-a77e020303b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:21.917496 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.917475 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76eb079-d83d-4467-873a-a77e020303b9-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:21.917571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.917499 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/549b99cf-bb7b-4d6e-b284-060f5a0d144d-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:21.956642 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:21.955998 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76eb079-d83d-4467-873a-a77e020303b9-kube-api-access-77845" pod "b76eb079-d83d-4467-873a-a77e020303b9" (UID: "b76eb079-d83d-4467-873a-a77e020303b9"). InnerVolumeSpecName "kube-api-access-77845". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:22.019186 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.019152 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77845\" (UniqueName: \"kubernetes.io/projected/b76eb079-d83d-4467-873a-a77e020303b9-kube-api-access-77845\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:22.047377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.047340 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549b99cf-bb7b-4d6e-b284-060f5a0d144d-kube-api-access-ctfb6" pod "549b99cf-bb7b-4d6e-b284-060f5a0d144d" (UID: "549b99cf-bb7b-4d6e-b284-060f5a0d144d"). InnerVolumeSpecName "kube-api-access-ctfb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:22.120730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.120698 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfb6\" (UniqueName: \"kubernetes.io/projected/549b99cf-bb7b-4d6e-b284-060f5a0d144d-kube-api-access-ctfb6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:22.363917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.362072 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" event={"ID":"b3869c86-0acc-4c84-9438-10f98b688ac8","Type":"ContainerDied","Data":"8554b3f02a82d4a4e2f585986707f1d775eb975e2c3f82432c52255f689969f0"} Nov 03 23:00:22.363917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.362142 4903 scope.go:122] "RemoveContainer" containerID="a42f6ad330aec73374e8b98a8fb58d4b4da05a3d5de8e824b17ec13b8840f210" Nov 03 23:00:22.363917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.362275 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v" Nov 03 23:00:22.368387 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.368242 4903 generic.go:356] "Generic (PLEG): container finished" podID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerID="09ecb303ffa3ee300337c40f9e002aeb82c7888101395184e1af84e26b7154f7" exitCode=60 Nov 03 23:00:22.368387 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.368347 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" event={"ID":"08fcf757-4727-4a59-8e4c-b07ad317df69","Type":"ContainerDied","Data":"09ecb303ffa3ee300337c40f9e002aeb82c7888101395184e1af84e26b7154f7"} Nov 03 23:00:22.368872 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.368826 4903 scope.go:122] "RemoveContainer" containerID="09ecb303ffa3ee300337c40f9e002aeb82c7888101395184e1af84e26b7154f7" Nov 03 23:00:22.369188 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:22.369159 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz_odh-ai-gateway-infra(08fcf757-4727-4a59-8e4c-b07ad317df69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" Nov 03 23:00:22.370353 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.370327 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" event={"ID":"549b99cf-bb7b-4d6e-b284-060f5a0d144d","Type":"ContainerDied","Data":"db1f471d73e3c5cf020dd2b0192d42a12cda9fee9db310b24a783e76ee086c25"} Nov 03 23:00:22.370353 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.370344 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm" Nov 03 23:00:22.375507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.375415 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" event={"ID":"b76eb079-d83d-4467-873a-a77e020303b9","Type":"ContainerDied","Data":"be237537d3917f606dd7254b095ccde85c577fffeedb1a23c037d3dd7a60dff4"} Nov 03 23:00:22.375507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.375455 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm" Nov 03 23:00:22.377510 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.377450 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" event={"ID":"dc69a099-02c2-45bd-aa5a-516e0680c0d2","Type":"ContainerDied","Data":"e02bd9e09df070fcd275df83b49942f4416a63bf0baa97fae13f782283c17040"} Nov 03 23:00:22.377510 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.377475 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr" Nov 03 23:00:22.380527 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.380447 4903 scope.go:122] "RemoveContainer" containerID="500a40c05cb1f8e8b322931939ec299e8f97ef1549da88fc9f9720fde1036525" Nov 03 23:00:22.383543 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.383523 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" Nov 03 23:00:22.383656 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.383579 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz" event={"ID":"55b2ad97-84d0-4500-8645-172ddb9b1aa4","Type":"ContainerDied","Data":"bb68e573675ddf5da15e723e885ff2abd34c922318cffb0f04c2cab7448d3d9c"} Nov 03 23:00:22.387382 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.387003 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" event={"ID":"e6f10255-ffed-4e46-b2b9-567ce601cc11","Type":"ContainerStarted","Data":"0e690b50160c88036746e962a3eab8be594a8e5e2fa427fd5ead4d1583cb9f31"} Nov 03 23:00:22.390052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.390031 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" event={"ID":"c4c07b85-9bec-46d6-a99c-8a703d94e303","Type":"ContainerStarted","Data":"c8bf4243aa4f52092f94ecd7feff68bff4f0edb27119d920dc0692ae1ea5ff7c"} Nov 03 23:00:22.393770 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.393743 4903 generic.go:356] "Generic (PLEG): container finished" podID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerID="840c864af532c3472828bc1566e027330e6ad1b72b0df797550dab4479de10ad" exitCode=60 Nov 03 23:00:22.393859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.393808 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" event={"ID":"d7f11ee4-b643-4fa5-b040-6e3532ee98a3","Type":"ContainerDied","Data":"840c864af532c3472828bc1566e027330e6ad1b72b0df797550dab4479de10ad"} Nov 03 23:00:22.394143 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.394129 4903 scope.go:122] "RemoveContainer" containerID="840c864af532c3472828bc1566e027330e6ad1b72b0df797550dab4479de10ad" Nov 03 23:00:22.394398 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:22.394374 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p_odh-ai-gateway-infra(d7f11ee4-b643-4fa5-b040-6e3532ee98a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" Nov 03 23:00:22.400246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.400224 4903 scope.go:122] "RemoveContainer" containerID="50ebdea83cd8e4d147b1b87d50b5c23a40aec62f6aa93faab1c9136d21964f3b" Nov 03 23:00:22.415197 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.415170 4903 scope.go:122] "RemoveContainer" containerID="bf749a2296c0074adf501f7bbd5aed747a023b200101b434aed768c2a5ade001" Nov 03 23:00:22.419664 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.419099 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v"] Nov 03 23:00:22.424720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.424697 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-9b841d-29895780-g9z2v"] Nov 03 23:00:22.453893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.449339 4903 scope.go:122] "RemoveContainer" containerID="40fe950fbed583b652691ce81a9de0e07c326cc6297ff906441d6523571833fd" Nov 03 23:00:22.473020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.473001 4903 scope.go:122] "RemoveContainer" containerID="59aa3bc0d669b4a412cd6330f99e6df1e33c75efda4ad158e9070493ab134509" Nov 03 23:00:22.488516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.488491 4903 scope.go:122] "RemoveContainer" containerID="ad3c9c6466cca0153627779cf0b49feb323fd0711b80600c5007e00eef4c55df" Nov 03 23:00:22.548881 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.545524 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p"] Nov 03 23:00:22.548881 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.546926 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm"] Nov 03 23:00:22.562224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.557222 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a940d5-29895780-ddrkm"] Nov 03 23:00:22.585408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.585373 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr"] Nov 03 23:00:22.590208 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.590182 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0847aa-29895780-78nvr"] Nov 03 23:00:22.613163 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.613136 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:22.626711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.626686 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390914dc-6339-4110-a8f0-d3c43479e328-kube-api-access-zrwgw\" (UniqueName: \"kubernetes.io/projected/390914dc-6339-4110-a8f0-d3c43479e328-kube-api-access-zrwgw\") pod \"390914dc-6339-4110-a8f0-d3c43479e328\" (UID: \"390914dc-6339-4110-a8f0-d3c43479e328\") " Nov 03 23:00:22.626862 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.626756 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390914dc-6339-4110-a8f0-d3c43479e328-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390914dc-6339-4110-a8f0-d3c43479e328-maas-api-service-ca\") pod \"390914dc-6339-4110-a8f0-d3c43479e328\" (UID: \"390914dc-6339-4110-a8f0-d3c43479e328\") " Nov 03 23:00:22.627234 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.627211 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390914dc-6339-4110-a8f0-d3c43479e328-maas-api-service-ca" pod "390914dc-6339-4110-a8f0-d3c43479e328" (UID: "390914dc-6339-4110-a8f0-d3c43479e328"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:22.629521 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.629500 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390914dc-6339-4110-a8f0-d3c43479e328-kube-api-access-zrwgw" pod "390914dc-6339-4110-a8f0-d3c43479e328" (UID: "390914dc-6339-4110-a8f0-d3c43479e328"). InnerVolumeSpecName "kube-api-access-zrwgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:22.659081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.659049 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz"] Nov 03 23:00:22.663608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.663584 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-26f8ba-29895780-dzkdz"] Nov 03 23:00:22.668152 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.668128 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz"] Nov 03 23:00:22.724650 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.724616 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm"] Nov 03 23:00:22.728196 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.728162 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrwgw\" (UniqueName: \"kubernetes.io/projected/390914dc-6339-4110-a8f0-d3c43479e328-kube-api-access-zrwgw\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:22.728196 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.728193 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/390914dc-6339-4110-a8f0-d3c43479e328-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:22.731597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:22.731571 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4f9bb0-29895780-mqzsm"] Nov 03 23:00:23.410074 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.410034 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" event={"ID":"390914dc-6339-4110-a8f0-d3c43479e328","Type":"ContainerDied","Data":"eb1e56cfab9385e996db0ce323835c0f9b0c60d73699d438ce3eac70f0c54892"} Nov 03 23:00:23.410216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.410127 4903 scope.go:122] "RemoveContainer" containerID="d2cbbb522d396d811d4e74e9a0b58e63064cd8728902441516b00ca02abf5bc0" Nov 03 23:00:23.410216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.410055 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw" Nov 03 23:00:23.419840 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.419804 4903 generic.go:356] "Generic (PLEG): container finished" podID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerID="5f9748e5cb0558e6f968ea78dae161560b47c754587ae799c0ecf5d015807aad" exitCode=60 Nov 03 23:00:23.419990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.419893 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" event={"ID":"bee443a3-007b-46c1-bdea-21ab7ea099db","Type":"ContainerDied","Data":"5f9748e5cb0558e6f968ea78dae161560b47c754587ae799c0ecf5d015807aad"} Nov 03 23:00:23.420385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.420364 4903 scope.go:122] "RemoveContainer" containerID="5f9748e5cb0558e6f968ea78dae161560b47c754587ae799c0ecf5d015807aad" Nov 03 23:00:23.422705 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:23.420783 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89_odh-ai-gateway-infra(bee443a3-007b-46c1-bdea-21ab7ea099db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" Nov 03 23:00:23.426772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.425926 4903 generic.go:356] "Generic (PLEG): container finished" podID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerID="31cc8503581f09bf8ad4608876d93e045d4946156a73644356cc261c24be854f" exitCode=60 Nov 03 23:00:23.426772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.426004 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" event={"ID":"42a165c7-65a1-4e2d-9b8b-10d962afbf1b","Type":"ContainerDied","Data":"31cc8503581f09bf8ad4608876d93e045d4946156a73644356cc261c24be854f"} Nov 03 23:00:23.426772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.426414 4903 scope.go:122] "RemoveContainer" containerID="31cc8503581f09bf8ad4608876d93e045d4946156a73644356cc261c24be854f" Nov 03 23:00:23.426772 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:23.426729 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5_odh-ai-gateway-infra(42a165c7-65a1-4e2d-9b8b-10d962afbf1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" Nov 03 23:00:23.427858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.427707 4903 scope.go:122] "RemoveContainer" containerID="1dad136232d2851b50846fddb733fbfb68fbf153b4ccdc733dd6669695dfc98b" Nov 03 23:00:23.434167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.434146 4903 generic.go:356] "Generic (PLEG): container finished" podID="ba749010-2688-407c-b125-f8f88bb73af1" containerID="1509a149915dd2d3f2b456e77cf048914b70a9cff24721c985b6a624ef364cb7" exitCode=60 Nov 03 23:00:23.434326 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.434224 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" event={"ID":"ba749010-2688-407c-b125-f8f88bb73af1","Type":"ContainerDied","Data":"1509a149915dd2d3f2b456e77cf048914b70a9cff24721c985b6a624ef364cb7"} Nov 03 23:00:23.434687 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.434671 4903 scope.go:122] "RemoveContainer" containerID="1509a149915dd2d3f2b456e77cf048914b70a9cff24721c985b6a624ef364cb7" Nov 03 23:00:23.435070 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:23.435051 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt_odh-ai-gateway-infra(ba749010-2688-407c-b125-f8f88bb73af1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" podUID="ba749010-2688-407c-b125-f8f88bb73af1" Nov 03 23:00:23.436820 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.436798 4903 generic.go:356] "Generic (PLEG): container finished" podID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerID="0e690b50160c88036746e962a3eab8be594a8e5e2fa427fd5ead4d1583cb9f31" exitCode=60 Nov 03 23:00:23.436908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.436875 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" event={"ID":"e6f10255-ffed-4e46-b2b9-567ce601cc11","Type":"ContainerDied","Data":"0e690b50160c88036746e962a3eab8be594a8e5e2fa427fd5ead4d1583cb9f31"} Nov 03 23:00:23.437902 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.437434 4903 scope.go:122] "RemoveContainer" containerID="0e690b50160c88036746e962a3eab8be594a8e5e2fa427fd5ead4d1583cb9f31" Nov 03 23:00:23.437902 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:23.437854 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb_odh-ai-gateway-infra(e6f10255-ffed-4e46-b2b9-567ce601cc11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" Nov 03 23:00:23.439744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.439674 4903 generic.go:356] "Generic (PLEG): container finished" podID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerID="c8bf4243aa4f52092f94ecd7feff68bff4f0edb27119d920dc0692ae1ea5ff7c" exitCode=60 Nov 03 23:00:23.439744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.439712 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" event={"ID":"c4c07b85-9bec-46d6-a99c-8a703d94e303","Type":"ContainerDied","Data":"c8bf4243aa4f52092f94ecd7feff68bff4f0edb27119d920dc0692ae1ea5ff7c"} Nov 03 23:00:23.440029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.440011 4903 scope.go:122] "RemoveContainer" containerID="c8bf4243aa4f52092f94ecd7feff68bff4f0edb27119d920dc0692ae1ea5ff7c" Nov 03 23:00:23.440237 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:23.440218 4903 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb_odh-ai-gateway-infra(c4c07b85-9bec-46d6-a99c-8a703d94e303)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" Nov 03 23:00:23.444680 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.444658 4903 scope.go:122] "RemoveContainer" containerID="797c2c9bfe43779fa5c8902465d183038f989ea93ccef981736ba86d20468e24" Nov 03 23:00:23.467233 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.467207 4903 scope.go:122] "RemoveContainer" containerID="88b61cc5d5c31cb56a2c8542070469f7d9da1860643dac5dc7d1d39114dff896" Nov 03 23:00:23.491878 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.490985 4903 scope.go:122] "RemoveContainer" containerID="c84b421e24fe73f42a10d7143460e977bbd3080023ed9ccfea64a703244ba8b2" Nov 03 23:00:23.523833 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.523805 4903 scope.go:122] "RemoveContainer" containerID="1651b0737259c5b9b3a535d862612149e86cca7d71ae23aa837e251f598f90bd" Nov 03 23:00:23.606798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.606769 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw"] Nov 03 23:00:23.612830 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.612805 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-b023a2-29895780-dkttw"] Nov 03 23:00:23.617219 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.617196 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:23.641237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.640961 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-maas-api-service-ca\") pod \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\" (UID: \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\") " Nov 03 23:00:23.641237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.641126 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-kube-api-access-xhqkg\" (UniqueName: \"kubernetes.io/projected/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-kube-api-access-xhqkg\") pod \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\" (UID: \"d7f11ee4-b643-4fa5-b040-6e3532ee98a3\") " Nov 03 23:00:23.641486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.641453 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-maas-api-service-ca" pod "d7f11ee4-b643-4fa5-b040-6e3532ee98a3" (UID: "d7f11ee4-b643-4fa5-b040-6e3532ee98a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:23.641784 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.641760 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:23.643402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.643373 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-kube-api-access-xhqkg" pod "d7f11ee4-b643-4fa5-b040-6e3532ee98a3" (UID: "d7f11ee4-b643-4fa5-b040-6e3532ee98a3"). InnerVolumeSpecName "kube-api-access-xhqkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:23.674942 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.674919 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:23.737210 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.737177 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390914dc-6339-4110-a8f0-d3c43479e328" path="/var/lib/kubelet/pods/390914dc-6339-4110-a8f0-d3c43479e328/volumes" Nov 03 23:00:23.737708 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.737692 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" path="/var/lib/kubelet/pods/549b99cf-bb7b-4d6e-b284-060f5a0d144d/volumes" Nov 03 23:00:23.738098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.738086 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" path="/var/lib/kubelet/pods/55b2ad97-84d0-4500-8645-172ddb9b1aa4/volumes" Nov 03 23:00:23.738822 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.738806 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" path="/var/lib/kubelet/pods/b3869c86-0acc-4c84-9438-10f98b688ac8/volumes" Nov 03 23:00:23.739205 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.739192 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76eb079-d83d-4467-873a-a77e020303b9" path="/var/lib/kubelet/pods/b76eb079-d83d-4467-873a-a77e020303b9/volumes" Nov 03 23:00:23.739636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.739617 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" path="/var/lib/kubelet/pods/dc69a099-02c2-45bd-aa5a-516e0680c0d2/volumes" Nov 03 23:00:23.742267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.742244 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fcf757-4727-4a59-8e4c-b07ad317df69-kube-api-access-hfktk\" (UniqueName: \"kubernetes.io/projected/08fcf757-4727-4a59-8e4c-b07ad317df69-kube-api-access-hfktk\") pod \"08fcf757-4727-4a59-8e4c-b07ad317df69\" (UID: \"08fcf757-4727-4a59-8e4c-b07ad317df69\") " Nov 03 23:00:23.742450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.742348 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fcf757-4727-4a59-8e4c-b07ad317df69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf757-4727-4a59-8e4c-b07ad317df69-maas-api-service-ca\") pod \"08fcf757-4727-4a59-8e4c-b07ad317df69\" (UID: \"08fcf757-4727-4a59-8e4c-b07ad317df69\") " Nov 03 23:00:23.742936 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.742911 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhqkg\" (UniqueName: \"kubernetes.io/projected/d7f11ee4-b643-4fa5-b040-6e3532ee98a3-kube-api-access-xhqkg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:23.743193 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.743169 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fcf757-4727-4a59-8e4c-b07ad317df69-maas-api-service-ca" pod "08fcf757-4727-4a59-8e4c-b07ad317df69" (UID: "08fcf757-4727-4a59-8e4c-b07ad317df69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:23.745177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.745152 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fcf757-4727-4a59-8e4c-b07ad317df69-kube-api-access-hfktk" pod "08fcf757-4727-4a59-8e4c-b07ad317df69" (UID: "08fcf757-4727-4a59-8e4c-b07ad317df69"). InnerVolumeSpecName "kube-api-access-hfktk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:23.778587 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.778560 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb"] Nov 03 23:00:23.814396 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.814367 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb"] Nov 03 23:00:23.843997 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.843964 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfktk\" (UniqueName: \"kubernetes.io/projected/08fcf757-4727-4a59-8e4c-b07ad317df69-kube-api-access-hfktk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:23.843997 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:23.843993 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcf757-4727-4a59-8e4c-b07ad317df69-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:24.457926 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.457897 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" Nov 03 23:00:24.457926 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.457909 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p" event={"ID":"d7f11ee4-b643-4fa5-b040-6e3532ee98a3","Type":"ContainerDied","Data":"f93ee5063fb3da6ca60ee3e38991d2097ed4232055393042b8e67a0d980e6396"} Nov 03 23:00:24.458106 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.458075 4903 scope.go:122] "RemoveContainer" containerID="840c864af532c3472828bc1566e027330e6ad1b72b0df797550dab4479de10ad" Nov 03 23:00:24.461055 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.461028 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" event={"ID":"08fcf757-4727-4a59-8e4c-b07ad317df69","Type":"ContainerDied","Data":"4b1ddf2be88ef323aec8d2abaf01e3310f89e832db5453562919ca6a23e64b7f"} Nov 03 23:00:24.461152 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.461138 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz" Nov 03 23:00:24.476193 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.476167 4903 scope.go:122] "RemoveContainer" containerID="09ecb303ffa3ee300337c40f9e002aeb82c7888101395184e1af84e26b7154f7" Nov 03 23:00:24.499606 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.499565 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt"] Nov 03 23:00:24.587410 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.587375 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p"] Nov 03 23:00:24.590827 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.590798 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5"] Nov 03 23:00:24.596666 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.596620 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-553753-29895780-r498p"] Nov 03 23:00:24.662367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.662336 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89"] Nov 03 23:00:24.669386 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.669348 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz"] Nov 03 23:00:24.676399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.674857 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d4b6bb-29895780-jr7bz"] Nov 03 23:00:24.698075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.695805 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:24.764405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.762602 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4c07b85-9bec-46d6-a99c-8a703d94e303-kube-api-access-fxww6\" (UniqueName: \"kubernetes.io/projected/c4c07b85-9bec-46d6-a99c-8a703d94e303-kube-api-access-fxww6\") pod \"c4c07b85-9bec-46d6-a99c-8a703d94e303\" (UID: \"c4c07b85-9bec-46d6-a99c-8a703d94e303\") " Nov 03 23:00:24.764405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.762788 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4c07b85-9bec-46d6-a99c-8a703d94e303-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07b85-9bec-46d6-a99c-8a703d94e303-maas-api-service-ca\") pod \"c4c07b85-9bec-46d6-a99c-8a703d94e303\" (UID: \"c4c07b85-9bec-46d6-a99c-8a703d94e303\") " Nov 03 23:00:24.764405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.763881 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c07b85-9bec-46d6-a99c-8a703d94e303-maas-api-service-ca" pod "c4c07b85-9bec-46d6-a99c-8a703d94e303" (UID: "c4c07b85-9bec-46d6-a99c-8a703d94e303"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:24.766906 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.766879 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c07b85-9bec-46d6-a99c-8a703d94e303-kube-api-access-fxww6" pod "c4c07b85-9bec-46d6-a99c-8a703d94e303" (UID: "c4c07b85-9bec-46d6-a99c-8a703d94e303"). InnerVolumeSpecName "kube-api-access-fxww6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:24.800104 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.800081 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:24.809426 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.809270 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:24.864318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.864013 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba749010-2688-407c-b125-f8f88bb73af1-kube-api-access-q5n79\" (UniqueName: \"kubernetes.io/projected/ba749010-2688-407c-b125-f8f88bb73af1-kube-api-access-q5n79\") pod \"ba749010-2688-407c-b125-f8f88bb73af1\" (UID: \"ba749010-2688-407c-b125-f8f88bb73af1\") " Nov 03 23:00:24.864318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.864125 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f10255-ffed-4e46-b2b9-567ce601cc11-kube-api-access-nxhv6\" (UniqueName: \"kubernetes.io/projected/e6f10255-ffed-4e46-b2b9-567ce601cc11-kube-api-access-nxhv6\") pod \"e6f10255-ffed-4e46-b2b9-567ce601cc11\" (UID: \"e6f10255-ffed-4e46-b2b9-567ce601cc11\") " Nov 03 23:00:24.864318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.864216 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba749010-2688-407c-b125-f8f88bb73af1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba749010-2688-407c-b125-f8f88bb73af1-maas-api-service-ca\") pod \"ba749010-2688-407c-b125-f8f88bb73af1\" (UID: \"ba749010-2688-407c-b125-f8f88bb73af1\") " Nov 03 23:00:24.864318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.864271 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f10255-ffed-4e46-b2b9-567ce601cc11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f10255-ffed-4e46-b2b9-567ce601cc11-maas-api-service-ca\") pod \"e6f10255-ffed-4e46-b2b9-567ce601cc11\" (UID: \"e6f10255-ffed-4e46-b2b9-567ce601cc11\") " Nov 03 23:00:24.864749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.864721 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba749010-2688-407c-b125-f8f88bb73af1-maas-api-service-ca" pod "ba749010-2688-407c-b125-f8f88bb73af1" (UID: "ba749010-2688-407c-b125-f8f88bb73af1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:24.865044 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.864915 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f10255-ffed-4e46-b2b9-567ce601cc11-maas-api-service-ca" pod "e6f10255-ffed-4e46-b2b9-567ce601cc11" (UID: "e6f10255-ffed-4e46-b2b9-567ce601cc11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:24.865141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.864917 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07b85-9bec-46d6-a99c-8a703d94e303-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:24.865249 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.865234 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxww6\" (UniqueName: \"kubernetes.io/projected/c4c07b85-9bec-46d6-a99c-8a703d94e303-kube-api-access-fxww6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:24.867721 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.867690 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f10255-ffed-4e46-b2b9-567ce601cc11-kube-api-access-nxhv6" pod "e6f10255-ffed-4e46-b2b9-567ce601cc11" (UID: "e6f10255-ffed-4e46-b2b9-567ce601cc11"). InnerVolumeSpecName "kube-api-access-nxhv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:24.869790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.869756 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba749010-2688-407c-b125-f8f88bb73af1-kube-api-access-q5n79" pod "ba749010-2688-407c-b125-f8f88bb73af1" (UID: "ba749010-2688-407c-b125-f8f88bb73af1"). InnerVolumeSpecName "kube-api-access-q5n79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:24.897592 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.897569 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:24.900052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.900029 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:24.966179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.966091 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-kube-api-access-c9pqp\" (UniqueName: \"kubernetes.io/projected/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-kube-api-access-c9pqp\") pod \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\" (UID: \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\") " Nov 03 23:00:24.966179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.966130 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee443a3-007b-46c1-bdea-21ab7ea099db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee443a3-007b-46c1-bdea-21ab7ea099db-maas-api-service-ca\") pod \"bee443a3-007b-46c1-bdea-21ab7ea099db\" (UID: \"bee443a3-007b-46c1-bdea-21ab7ea099db\") " Nov 03 23:00:24.966179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.966161 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee443a3-007b-46c1-bdea-21ab7ea099db-kube-api-access-hjmns\" (UniqueName: \"kubernetes.io/projected/bee443a3-007b-46c1-bdea-21ab7ea099db-kube-api-access-hjmns\") pod \"bee443a3-007b-46c1-bdea-21ab7ea099db\" (UID: \"bee443a3-007b-46c1-bdea-21ab7ea099db\") " Nov 03 23:00:24.966504 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.966477 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-maas-api-service-ca\") pod \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\" (UID: \"42a165c7-65a1-4e2d-9b8b-10d962afbf1b\") " Nov 03 23:00:24.966623 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.966593 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee443a3-007b-46c1-bdea-21ab7ea099db-maas-api-service-ca" pod "bee443a3-007b-46c1-bdea-21ab7ea099db" (UID: "bee443a3-007b-46c1-bdea-21ab7ea099db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:24.966864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.966837 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-maas-api-service-ca" pod "42a165c7-65a1-4e2d-9b8b-10d962afbf1b" (UID: "42a165c7-65a1-4e2d-9b8b-10d962afbf1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:24.967516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.967493 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:24.967636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.967519 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5n79\" (UniqueName: \"kubernetes.io/projected/ba749010-2688-407c-b125-f8f88bb73af1-kube-api-access-q5n79\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:24.967636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.967533 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee443a3-007b-46c1-bdea-21ab7ea099db-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:24.967636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.967548 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxhv6\" (UniqueName: \"kubernetes.io/projected/e6f10255-ffed-4e46-b2b9-567ce601cc11-kube-api-access-nxhv6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:24.967636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.967561 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba749010-2688-407c-b125-f8f88bb73af1-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:24.967636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.967575 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f10255-ffed-4e46-b2b9-567ce601cc11-maas-api-service-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:24.968689 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.968660 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee443a3-007b-46c1-bdea-21ab7ea099db-kube-api-access-hjmns" pod "bee443a3-007b-46c1-bdea-21ab7ea099db" (UID: "bee443a3-007b-46c1-bdea-21ab7ea099db"). InnerVolumeSpecName "kube-api-access-hjmns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:24.968917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:24.968895 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-kube-api-access-c9pqp" pod "42a165c7-65a1-4e2d-9b8b-10d962afbf1b" (UID: "42a165c7-65a1-4e2d-9b8b-10d962afbf1b"). InnerVolumeSpecName "kube-api-access-c9pqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:25.069637 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.069607 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pqp\" (UniqueName: \"kubernetes.io/projected/42a165c7-65a1-4e2d-9b8b-10d962afbf1b-kube-api-access-c9pqp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:25.069637 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.069641 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjmns\" (UniqueName: \"kubernetes.io/projected/bee443a3-007b-46c1-bdea-21ab7ea099db-kube-api-access-hjmns\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:25.471995 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.471960 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" Nov 03 23:00:25.471995 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.471987 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt" event={"ID":"ba749010-2688-407c-b125-f8f88bb73af1","Type":"ContainerDied","Data":"57585a47c736f6ff67a2aeee2dda22732c998e1f46ac3e7ed45a62b7e6408cc4"} Nov 03 23:00:25.472269 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.472042 4903 scope.go:122] "RemoveContainer" containerID="1509a149915dd2d3f2b456e77cf048914b70a9cff24721c985b6a624ef364cb7" Nov 03 23:00:25.473803 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.473776 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" Nov 03 23:00:25.473803 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.473788 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb" event={"ID":"e6f10255-ffed-4e46-b2b9-567ce601cc11","Type":"ContainerDied","Data":"af211dcdd35243578f3b099fc23201b1726c1105b98e9c9b9c6edbbc1e70880d"} Nov 03 23:00:25.482165 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.482139 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" event={"ID":"c4c07b85-9bec-46d6-a99c-8a703d94e303","Type":"ContainerDied","Data":"7768fe2d83b0ef0aa77cedea84dd8530c3d8be9d143d1d97b47a3c72162766ba"} Nov 03 23:00:25.482259 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.482126 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb" Nov 03 23:00:25.484519 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.484500 4903 scope.go:122] "RemoveContainer" containerID="0e690b50160c88036746e962a3eab8be594a8e5e2fa427fd5ead4d1583cb9f31" Nov 03 23:00:25.486612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.486594 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" Nov 03 23:00:25.486612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.486602 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89" event={"ID":"bee443a3-007b-46c1-bdea-21ab7ea099db","Type":"ContainerDied","Data":"4e6952cfaf300a8377579f0dfd20c214f5a3f12ba1695e139255bdb4eb7d2b3c"} Nov 03 23:00:25.488220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.488199 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" Nov 03 23:00:25.488220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.488216 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5" event={"ID":"42a165c7-65a1-4e2d-9b8b-10d962afbf1b","Type":"ContainerDied","Data":"d9ef5fd51c12f942ac7b87f6ed8249afa52a8c902895a1789dbc07321dedaa5b"} Nov 03 23:00:25.525506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.525481 4903 scope.go:122] "RemoveContainer" containerID="c8bf4243aa4f52092f94ecd7feff68bff4f0edb27119d920dc0692ae1ea5ff7c" Nov 03 23:00:25.526203 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.526177 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt"] Nov 03 23:00:25.534093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.534066 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2d619f-29895780-cmfvt"] Nov 03 23:00:25.538001 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.537967 4903 scope.go:122] "RemoveContainer" containerID="5f9748e5cb0558e6f968ea78dae161560b47c754587ae799c0ecf5d015807aad" Nov 03 23:00:25.551162 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.551142 4903 scope.go:122] "RemoveContainer" containerID="31cc8503581f09bf8ad4608876d93e045d4946156a73644356cc261c24be854f" Nov 03 23:00:25.604190 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.604153 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb"] Nov 03 23:00:25.613372 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.613339 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-68410c-29895780-npxdb"] Nov 03 23:00:25.697929 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.697887 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5"] Nov 03 23:00:25.720650 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.720612 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d8a3d1-29895780-765m5"] Nov 03 23:00:25.736482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.736409 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" path="/var/lib/kubelet/pods/08fcf757-4727-4a59-8e4c-b07ad317df69/volumes" Nov 03 23:00:25.736892 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.736876 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" path="/var/lib/kubelet/pods/42a165c7-65a1-4e2d-9b8b-10d962afbf1b/volumes" Nov 03 23:00:25.737268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.737256 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba749010-2688-407c-b125-f8f88bb73af1" path="/var/lib/kubelet/pods/ba749010-2688-407c-b125-f8f88bb73af1/volumes" Nov 03 23:00:25.737997 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.737982 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" path="/var/lib/kubelet/pods/d7f11ee4-b643-4fa5-b040-6e3532ee98a3/volumes" Nov 03 23:00:25.738397 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.738384 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" path="/var/lib/kubelet/pods/e6f10255-ffed-4e46-b2b9-567ce601cc11/volumes" Nov 03 23:00:25.753744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.753715 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89"] Nov 03 23:00:25.757666 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.757646 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-15e7a7-29895780-hpm89"] Nov 03 23:00:25.786153 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.786123 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb"] Nov 03 23:00:25.794989 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:25.794962 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-434ce3-29895780-fz7pb"] Nov 03 23:00:27.736935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:27.736900 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" path="/var/lib/kubelet/pods/bee443a3-007b-46c1-bdea-21ab7ea099db/volumes" Nov 03 23:00:27.737545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:27.737528 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" path="/var/lib/kubelet/pods/c4c07b85-9bec-46d6-a99c-8a703d94e303/volumes" Nov 03 23:00:28.289314 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.286112 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx"] Nov 03 23:00:28.291496 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291468 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerName="cleanup" Nov 03 23:00:28.291496 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291494 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291509 4903 cpu_manager.go:414] "containerMap: removing container" podUID="390914dc-6339-4110-a8f0-d3c43479e328" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291518 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390914dc-6339-4110-a8f0-d3c43479e328" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291532 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291539 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291553 4903 cpu_manager.go:414] "containerMap: removing container" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291559 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291569 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291575 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291589 4903 cpu_manager.go:414] "containerMap: removing container" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291595 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291603 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291609 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291620 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ba749010-2688-407c-b125-f8f88bb73af1" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291626 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba749010-2688-407c-b125-f8f88bb73af1" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291640 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291646 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291655 4903 cpu_manager.go:414] "containerMap: removing container" podUID="390914dc-6339-4110-a8f0-d3c43479e328" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291661 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390914dc-6339-4110-a8f0-d3c43479e328" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291673 4903 cpu_manager.go:414] "containerMap: removing container" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerName="cleanup" Nov 03 23:00:28.291670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291679 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291689 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291695 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291705 4903 cpu_manager.go:414] "containerMap: removing container" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291711 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291720 4903 cpu_manager.go:414] "containerMap: removing container" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291727 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291738 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b76eb079-d83d-4467-873a-a77e020303b9" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291744 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76eb079-d83d-4467-873a-a77e020303b9" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291754 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291760 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291769 4903 cpu_manager.go:414] "containerMap: removing container" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291775 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291786 4903 cpu_manager.go:414] "containerMap: removing container" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291792 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291801 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291807 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291817 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291823 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291831 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291837 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291848 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291854 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291864 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291870 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291880 4903 cpu_manager.go:414] "containerMap: removing container" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291886 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291899 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291905 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291918 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291925 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291938 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b76eb079-d83d-4467-873a-a77e020303b9" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291945 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76eb079-d83d-4467-873a-a77e020303b9" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291953 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291959 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291973 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.291978 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292003 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292010 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292018 4903 cpu_manager.go:414] "containerMap: removing container" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292025 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292037 4903 cpu_manager.go:414] "containerMap: removing container" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292042 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292055 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a3e5ea2b-b9dc-469e-a43b-8e390dd573c2" containerName="revoke-keys" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292061 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e5ea2b-b9dc-469e-a43b-8e390dd573c2" containerName="revoke-keys" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292073 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ba749010-2688-407c-b125-f8f88bb73af1" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292079 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba749010-2688-407c-b125-f8f88bb73af1" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292088 4903 cpu_manager.go:414] "containerMap: removing container" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292094 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292107 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292114 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292129 4903 cpu_manager.go:414] "containerMap: removing container" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292135 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292146 4903 cpu_manager.go:414] "containerMap: removing container" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292152 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292349 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="390914dc-6339-4110-a8f0-d3c43479e328" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292362 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292370 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292379 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292389 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292397 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292404 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba749010-2688-407c-b125-f8f88bb73af1" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292411 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292418 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="390914dc-6339-4110-a8f0-d3c43479e328" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292428 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292438 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerName="cleanup" Nov 03 23:00:28.292399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292446 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292456 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76eb079-d83d-4467-873a-a77e020303b9" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292462 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292472 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b2ad97-84d0-4500-8645-172ddb9b1aa4" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292480 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292487 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292493 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292503 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292510 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c07b85-9bec-46d6-a99c-8a703d94e303" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292519 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292528 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292536 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292543 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9dda56-7458-454f-b28a-7b1122f5deed" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292552 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292559 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292569 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292578 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292585 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292592 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292602 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76eb079-d83d-4467-873a-a77e020303b9" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292609 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292617 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e5ea2b-b9dc-469e-a43b-8e390dd573c2" containerName="revoke-keys" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292625 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba749010-2688-407c-b125-f8f88bb73af1" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292633 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292640 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292649 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerName="cleanup" Nov 03 23:00:28.295142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.292655 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" containerName="cleanup" Nov 03 23:00:28.300758 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.300730 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:28.303126 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.303100 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx"] Nov 03 23:00:28.315975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.315947 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-15e7a7\"" Nov 03 23:00:28.336907 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.336877 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-15e7a7-dockercfg-xw2rt\"" Nov 03 23:00:28.416766 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.416723 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8r79\" (UniqueName: \"kubernetes.io/projected/83ed7ec5-6b47-4934-9ef6-135fda1e2557-kube-api-access-h8r79\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx\" (UID: \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:28.417027 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.416998 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ed7ec5-6b47-4934-9ef6-135fda1e2557-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx\" (UID: \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:28.464410 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.464376 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m"] Nov 03 23:00:28.465337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465311 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerName="cleanup" Nov 03 23:00:28.465337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465333 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465357 4903 cpu_manager.go:414] "containerMap: removing container" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465367 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465381 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465388 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465412 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465420 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465437 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b76eb079-d83d-4467-873a-a77e020303b9" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465444 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76eb079-d83d-4467-873a-a77e020303b9" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465474 4903 cpu_manager.go:414] "containerMap: removing container" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465481 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549b99cf-bb7b-4d6e-b284-060f5a0d144d" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465493 4903 cpu_manager.go:414] "containerMap: removing container" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465499 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465529 4903 cpu_manager.go:414] "containerMap: removing container" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465536 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc69a099-02c2-45bd-aa5a-516e0680c0d2" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465570 4903 cpu_manager.go:414] "containerMap: removing container" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465578 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465588 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ba749010-2688-407c-b125-f8f88bb73af1" containerName="cleanup" Nov 03 23:00:28.465591 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465595 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba749010-2688-407c-b125-f8f88bb73af1" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465611 4903 cpu_manager.go:414] "containerMap: removing container" podUID="390914dc-6339-4110-a8f0-d3c43479e328" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465618 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390914dc-6339-4110-a8f0-d3c43479e328" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465797 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee443a3-007b-46c1-bdea-21ab7ea099db" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465812 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76eb079-d83d-4467-873a-a77e020303b9" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465831 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a165c7-65a1-4e2d-9b8b-10d962afbf1b" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465840 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fcf757-4727-4a59-8e4c-b07ad317df69" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465850 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f10255-ffed-4e46-b2b9-567ce601cc11" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465859 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba749010-2688-407c-b125-f8f88bb73af1" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465868 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3869c86-0acc-4c84-9438-10f98b688ac8" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465879 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="390914dc-6339-4110-a8f0-d3c43479e328" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465890 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74ad0b8-94d4-4fbf-9443-d362d91bec33" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465899 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="933ae1de-a9e1-44da-b47d-9d6177640711" containerName="cleanup" Nov 03 23:00:28.466484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.465915 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f11ee4-b643-4fa5-b040-6e3532ee98a3" containerName="cleanup" Nov 03 23:00:28.470425 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.470400 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:28.488769 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.488740 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m"] Nov 03 23:00:28.517942 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.517904 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ed7ec5-6b47-4934-9ef6-135fda1e2557-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx\" (UID: \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:28.518158 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.518028 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8r79\" (UniqueName: \"kubernetes.io/projected/83ed7ec5-6b47-4934-9ef6-135fda1e2557-kube-api-access-h8r79\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx\" (UID: \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:28.518158 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.518129 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77mk\" (UniqueName: \"kubernetes.io/projected/8a3373dd-b15e-490d-9a62-8b271356e69f-kube-api-access-f77mk\") pod \"payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m\" (UID: \"8a3373dd-b15e-490d-9a62-8b271356e69f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:28.518275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.518193 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3373dd-b15e-490d-9a62-8b271356e69f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m\" (UID: \"8a3373dd-b15e-490d-9a62-8b271356e69f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:28.518772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.518752 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ed7ec5-6b47-4934-9ef6-135fda1e2557-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx\" (UID: \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:28.540572 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.540495 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8r79\" (UniqueName: \"kubernetes.io/projected/83ed7ec5-6b47-4934-9ef6-135fda1e2557-kube-api-access-h8r79\") pod \"payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx\" (UID: \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:28.619661 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.619623 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3373dd-b15e-490d-9a62-8b271356e69f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m\" (UID: \"8a3373dd-b15e-490d-9a62-8b271356e69f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:28.619971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.619948 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77mk\" (UniqueName: \"kubernetes.io/projected/8a3373dd-b15e-490d-9a62-8b271356e69f-kube-api-access-f77mk\") pod \"payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m\" (UID: \"8a3373dd-b15e-490d-9a62-8b271356e69f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:28.620527 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.620504 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3373dd-b15e-490d-9a62-8b271356e69f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m\" (UID: \"8a3373dd-b15e-490d-9a62-8b271356e69f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:28.641386 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.641352 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77mk\" (UniqueName: \"kubernetes.io/projected/8a3373dd-b15e-490d-9a62-8b271356e69f-kube-api-access-f77mk\") pod \"payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m\" (UID: \"8a3373dd-b15e-490d-9a62-8b271356e69f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:28.647111 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.647085 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:28.789121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.789086 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:28.962866 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:28.962836 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx"] Nov 03 23:00:28.962975 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:28.962934 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ed7ec5_6b47_4934_9ef6_135fda1e2557.slice/crio-23626a07369d69ac72dfa07f3d8fed1d506abf53c84eb888b0b84ec1ac350973 WatchSource:0}: Error finding container 23626a07369d69ac72dfa07f3d8fed1d506abf53c84eb888b0b84ec1ac350973: Status 404 returned error can't find the container with id 23626a07369d69ac72dfa07f3d8fed1d506abf53c84eb888b0b84ec1ac350973 Nov 03 23:00:29.103976 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:29.103948 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m"] Nov 03 23:00:29.104572 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:29.104548 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3373dd_b15e_490d_9a62_8b271356e69f.slice/crio-e08bc4d9820cf52009fe5b0305f7b9d377b6fb30e9d18ec0941fc95a3bbceacc WatchSource:0}: Error finding container e08bc4d9820cf52009fe5b0305f7b9d377b6fb30e9d18ec0941fc95a3bbceacc: Status 404 returned error can't find the container with id e08bc4d9820cf52009fe5b0305f7b9d377b6fb30e9d18ec0941fc95a3bbceacc Nov 03 23:00:29.538403 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:29.538349 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" event={"ID":"8a3373dd-b15e-490d-9a62-8b271356e69f","Type":"ContainerStarted","Data":"335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096"} Nov 03 23:00:29.538632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:29.538417 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" event={"ID":"8a3373dd-b15e-490d-9a62-8b271356e69f","Type":"ContainerStarted","Data":"e08bc4d9820cf52009fe5b0305f7b9d377b6fb30e9d18ec0941fc95a3bbceacc"} Nov 03 23:00:29.538731 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:29.538682 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:29.540584 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:29.540557 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" event={"ID":"83ed7ec5-6b47-4934-9ef6-135fda1e2557","Type":"ContainerStarted","Data":"9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f"} Nov 03 23:00:29.540710 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:29.540604 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" event={"ID":"83ed7ec5-6b47-4934-9ef6-135fda1e2557","Type":"ContainerStarted","Data":"23626a07369d69ac72dfa07f3d8fed1d506abf53c84eb888b0b84ec1ac350973"} Nov 03 23:00:29.541098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:29.541079 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:29.576446 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:29.576404 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" podStartSLOduration=1.5763929490000002 podStartE2EDuration="1.576392949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:29.575611061 +0000 UTC m=+2855.979552036" watchObservedRunningTime="2026-11-03 23:00:29.576392949 +0000 UTC m=+2855.980333923" Nov 03 23:00:29.616444 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:29.616395 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" podStartSLOduration=1.616381523 podStartE2EDuration="1.616381523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:29.612147152 +0000 UTC m=+2856.016088127" watchObservedRunningTime="2026-11-03 23:00:29.616381523 +0000 UTC m=+2856.020322491" Nov 03 23:00:31.917014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.916973 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx"] Nov 03 23:00:31.921961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.921937 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:31.937543 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.937513 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:31.947676 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.947558 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:00:31.947676 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.947637 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:00:31.947676 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.947666 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:00:31.953068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.953040 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx"] Nov 03 23:00:31.982899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.982857 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9da02c3-nqwgx\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:31.983043 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.983021 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72d4\" (UniqueName: \"kubernetes.io/projected/f2e15a33-1307-4365-bc54-4435aa1b57d1-kube-api-access-w72d4\") pod \"maas-api-revoke-sub-d9da02c3-nqwgx\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:31.983139 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:31.983125 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9da02c3-nqwgx\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:32.084340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:32.084307 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9da02c3-nqwgx\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:32.084516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:32.084415 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72d4\" (UniqueName: \"kubernetes.io/projected/f2e15a33-1307-4365-bc54-4435aa1b57d1-kube-api-access-w72d4\") pod \"maas-api-revoke-sub-d9da02c3-nqwgx\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:32.084516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:32.084492 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9da02c3-nqwgx\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:32.085129 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:32.085103 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9da02c3-nqwgx\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:32.086708 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:32.086682 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9da02c3-nqwgx\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:32.105655 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:32.105632 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72d4\" (UniqueName: \"kubernetes.io/projected/f2e15a33-1307-4365-bc54-4435aa1b57d1-kube-api-access-w72d4\") pod \"maas-api-revoke-sub-d9da02c3-nqwgx\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:32.237652 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:32.237570 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:32.547856 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:32.547824 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx"] Nov 03 23:00:32.548284 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:32.548259 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e15a33_1307_4365_bc54_4435aa1b57d1.slice/crio-f7f262fb9105bc1c91690633eb5e5288728b2e5cb8568422e855b92d347610c7 WatchSource:0}: Error finding container f7f262fb9105bc1c91690633eb5e5288728b2e5cb8568422e855b92d347610c7: Status 404 returned error can't find the container with id f7f262fb9105bc1c91690633eb5e5288728b2e5cb8568422e855b92d347610c7 Nov 03 23:00:32.570915 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:32.570882 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" event={"ID":"f2e15a33-1307-4365-bc54-4435aa1b57d1","Type":"ContainerStarted","Data":"f7f262fb9105bc1c91690633eb5e5288728b2e5cb8568422e855b92d347610c7"} Nov 03 23:00:33.581211 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:33.581178 4903 generic.go:356] "Generic (PLEG): container finished" podID="f2e15a33-1307-4365-bc54-4435aa1b57d1" containerID="36334ffdf4ba75fb3f12938064e8b014923965573a4f437bd1b67ebe12c79747" exitCode=0 Nov 03 23:00:33.581641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:33.581267 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" event={"ID":"f2e15a33-1307-4365-bc54-4435aa1b57d1","Type":"ContainerDied","Data":"36334ffdf4ba75fb3f12938064e8b014923965573a4f437bd1b67ebe12c79747"} Nov 03 23:00:34.778679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:34.778649 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:34.922178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:34.922144 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e15a33-1307-4365-bc54-4435aa1b57d1-kube-api-access-w72d4\" (UniqueName: \"kubernetes.io/projected/f2e15a33-1307-4365-bc54-4435aa1b57d1-kube-api-access-w72d4\") pod \"f2e15a33-1307-4365-bc54-4435aa1b57d1\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " Nov 03 23:00:34.922387 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:34.922208 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-openshift-ca\") pod \"f2e15a33-1307-4365-bc54-4435aa1b57d1\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " Nov 03 23:00:34.922433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:34.922385 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-xks-ca\") pod \"f2e15a33-1307-4365-bc54-4435aa1b57d1\" (UID: \"f2e15a33-1307-4365-bc54-4435aa1b57d1\") " Nov 03 23:00:34.922645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:34.922619 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-openshift-ca" pod "f2e15a33-1307-4365-bc54-4435aa1b57d1" (UID: "f2e15a33-1307-4365-bc54-4435aa1b57d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:34.922923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:34.922903 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:34.924449 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:34.924427 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e15a33-1307-4365-bc54-4435aa1b57d1-kube-api-access-w72d4" pod "f2e15a33-1307-4365-bc54-4435aa1b57d1" (UID: "f2e15a33-1307-4365-bc54-4435aa1b57d1"). InnerVolumeSpecName "kube-api-access-w72d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:34.924449 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:34.924432 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-xks-ca" pod "f2e15a33-1307-4365-bc54-4435aa1b57d1" (UID: "f2e15a33-1307-4365-bc54-4435aa1b57d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:35.024367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:35.024330 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72d4\" (UniqueName: \"kubernetes.io/projected/f2e15a33-1307-4365-bc54-4435aa1b57d1-kube-api-access-w72d4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:35.024367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:35.024357 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e15a33-1307-4365-bc54-4435aa1b57d1-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:35.600151 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:35.600125 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" Nov 03 23:00:35.600151 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:35.600133 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx" event={"ID":"f2e15a33-1307-4365-bc54-4435aa1b57d1","Type":"ContainerDied","Data":"f7f262fb9105bc1c91690633eb5e5288728b2e5cb8568422e855b92d347610c7"} Nov 03 23:00:35.600401 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:35.600177 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7f262fb9105bc1c91690633eb5e5288728b2e5cb8568422e855b92d347610c7" Nov 03 23:00:38.391342 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.391308 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75"] Nov 03 23:00:38.392419 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.392386 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f2e15a33-1307-4365-bc54-4435aa1b57d1" containerName="revoke-keys" Nov 03 23:00:38.392419 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.392410 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e15a33-1307-4365-bc54-4435aa1b57d1" containerName="revoke-keys" Nov 03 23:00:38.392664 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.392640 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e15a33-1307-4365-bc54-4435aa1b57d1" containerName="revoke-keys" Nov 03 23:00:38.396732 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.396709 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.416079 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.416051 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75"] Nov 03 23:00:38.420079 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.420051 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-23d281\"/\"openshift-service-ca.crt\"" Nov 03 23:00:38.420079 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.420066 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-23d281\"/\"default-dockercfg-xckd5\"" Nov 03 23:00:38.420250 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.420149 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-23d281\"/\"rate-test-model-23d281-kserve-self-signed-certs\"" Nov 03 23:00:38.420250 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.420207 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-23d281\"/\"kube-root-ca.crt\"" Nov 03 23:00:38.477954 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.477924 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tls-certs\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.478118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.478080 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-dshm\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.478163 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.478115 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-home\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.478219 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.478199 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56wl9\" (UniqueName: \"kubernetes.io/projected/08a1b497-df74-4ee0-aa16-447d4a1f4c06-kube-api-access-56wl9\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.478304 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.478274 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tmp-dir\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.478396 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.478380 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-model-cache\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.580261 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.580226 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-model-cache\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.580445 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.580372 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tls-certs\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.580521 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.580495 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-dshm\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.580628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.580550 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-home\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.580701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.580653 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56wl9\" (UniqueName: \"kubernetes.io/projected/08a1b497-df74-4ee0-aa16-447d4a1f4c06-kube-api-access-56wl9\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.580701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.580695 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tmp-dir\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.580803 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.580728 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-model-cache\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.580933 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.580906 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-home\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.581007 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.580962 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tmp-dir\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.582777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.582750 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-dshm\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.582983 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.582965 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tls-certs\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.604335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.604308 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56wl9\" (UniqueName: \"kubernetes.io/projected/08a1b497-df74-4ee0-aa16-447d4a1f4c06-kube-api-access-56wl9\") pod \"rate-test-model-23d281-kserve-78865749-htr75\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.641223 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.641193 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr"] Nov 03 23:00:38.641461 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.641413 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" podUID="147459de-4415-4d95-b2a8-c3de6e4ecc8e" containerName="payload-processing" containerID="cri-o://d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43" gracePeriod=30 Nov 03 23:00:38.663327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.663278 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z"] Nov 03 23:00:38.663614 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.663559 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" podUID="05575897-8937-4040-97ea-3433a962f53b" containerName="payload-pre-processing" containerID="cri-o://1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80" gracePeriod=30 Nov 03 23:00:38.720792 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.720720 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:38.987612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:38.987588 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 23:00:39.020125 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.020080 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 23:00:39.086331 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:39.072642 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08a1b497_df74_4ee0_aa16_447d4a1f4c06.slice/crio-793765f5122fbce2eb5998f43841429cfea5cfdf669eae1e093ff2c8fafbbbae WatchSource:0}: Error finding container 793765f5122fbce2eb5998f43841429cfea5cfdf669eae1e093ff2c8fafbbbae: Status 404 returned error can't find the container with id 793765f5122fbce2eb5998f43841429cfea5cfdf669eae1e093ff2c8fafbbbae Nov 03 23:00:39.097421 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.089135 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147459de-4415-4d95-b2a8-c3de6e4ecc8e-kube-api-access-twcl7\" (UniqueName: \"kubernetes.io/projected/147459de-4415-4d95-b2a8-c3de6e4ecc8e-kube-api-access-twcl7\") pod \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\" (UID: \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\") " Nov 03 23:00:39.097421 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.089331 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147459de-4415-4d95-b2a8-c3de6e4ecc8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147459de-4415-4d95-b2a8-c3de6e4ecc8e-plugins-config-volume\") pod \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\" (UID: \"147459de-4415-4d95-b2a8-c3de6e4ecc8e\") " Nov 03 23:00:39.097421 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.097366 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147459de-4415-4d95-b2a8-c3de6e4ecc8e-plugins-config-volume" pod "147459de-4415-4d95-b2a8-c3de6e4ecc8e" (UID: "147459de-4415-4d95-b2a8-c3de6e4ecc8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:39.107892 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.107856 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147459de-4415-4d95-b2a8-c3de6e4ecc8e-kube-api-access-twcl7" pod "147459de-4415-4d95-b2a8-c3de6e4ecc8e" (UID: "147459de-4415-4d95-b2a8-c3de6e4ecc8e"). InnerVolumeSpecName "kube-api-access-twcl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:39.152635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.151340 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75"] Nov 03 23:00:39.191448 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.191412 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05575897-8937-4040-97ea-3433a962f53b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05575897-8937-4040-97ea-3433a962f53b-plugins-config-volume\") pod \"05575897-8937-4040-97ea-3433a962f53b\" (UID: \"05575897-8937-4040-97ea-3433a962f53b\") " Nov 03 23:00:39.191668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.191537 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05575897-8937-4040-97ea-3433a962f53b-kube-api-access-85rwd\" (UniqueName: \"kubernetes.io/projected/05575897-8937-4040-97ea-3433a962f53b-kube-api-access-85rwd\") pod \"05575897-8937-4040-97ea-3433a962f53b\" (UID: \"05575897-8937-4040-97ea-3433a962f53b\") " Nov 03 23:00:39.192433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.192123 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twcl7\" (UniqueName: \"kubernetes.io/projected/147459de-4415-4d95-b2a8-c3de6e4ecc8e-kube-api-access-twcl7\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:39.192433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.192148 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147459de-4415-4d95-b2a8-c3de6e4ecc8e-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:39.192882 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.192856 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05575897-8937-4040-97ea-3433a962f53b-plugins-config-volume" pod "05575897-8937-4040-97ea-3433a962f53b" (UID: "05575897-8937-4040-97ea-3433a962f53b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:39.194214 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.194189 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05575897-8937-4040-97ea-3433a962f53b-kube-api-access-85rwd" pod "05575897-8937-4040-97ea-3433a962f53b" (UID: "05575897-8937-4040-97ea-3433a962f53b"). InnerVolumeSpecName "kube-api-access-85rwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:39.270392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.270358 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v"] Nov 03 23:00:39.274919 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.274882 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-68410c-2444v"] Nov 03 23:00:39.293388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.293359 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05575897-8937-4040-97ea-3433a962f53b-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:39.293388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.293385 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rwd\" (UniqueName: \"kubernetes.io/projected/05575897-8937-4040-97ea-3433a962f53b-kube-api-access-85rwd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:39.634954 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.634922 4903 generic.go:356] "Generic (PLEG): container finished" podID="05575897-8937-4040-97ea-3433a962f53b" containerID="1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80" exitCode=0 Nov 03 23:00:39.635447 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.634963 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" Nov 03 23:00:39.635447 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.635005 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" event={"ID":"05575897-8937-4040-97ea-3433a962f53b","Type":"ContainerDied","Data":"1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80"} Nov 03 23:00:39.635447 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.635057 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z" event={"ID":"05575897-8937-4040-97ea-3433a962f53b","Type":"ContainerDied","Data":"3708ac75cac0ba0b7855991818f81e1b6b680eeb36271afca01501ea55716567"} Nov 03 23:00:39.635447 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.635088 4903 scope.go:122] "RemoveContainer" containerID="1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80" Nov 03 23:00:39.637111 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.637074 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" event={"ID":"08a1b497-df74-4ee0-aa16-447d4a1f4c06","Type":"ContainerStarted","Data":"c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0"} Nov 03 23:00:39.637241 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.637113 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" event={"ID":"08a1b497-df74-4ee0-aa16-447d4a1f4c06","Type":"ContainerStarted","Data":"793765f5122fbce2eb5998f43841429cfea5cfdf669eae1e093ff2c8fafbbbae"} Nov 03 23:00:39.638602 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.638581 4903 generic.go:356] "Generic (PLEG): container finished" podID="147459de-4415-4d95-b2a8-c3de6e4ecc8e" containerID="d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43" exitCode=0 Nov 03 23:00:39.638733 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.638639 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" Nov 03 23:00:39.638733 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.638663 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" event={"ID":"147459de-4415-4d95-b2a8-c3de6e4ecc8e","Type":"ContainerDied","Data":"d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43"} Nov 03 23:00:39.638733 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.638706 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr" event={"ID":"147459de-4415-4d95-b2a8-c3de6e4ecc8e","Type":"ContainerDied","Data":"712a5ae4b3e4d48a71e4552eae8808ded7b8ab9ff4ed3fc23cf40cc00e5fc6b5"} Nov 03 23:00:39.648146 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.647922 4903 scope.go:122] "RemoveContainer" containerID="1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80" Nov 03 23:00:39.648528 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:39.648334 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80\": container with ID starting with 1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80 not found: ID does not exist" containerID="1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80" Nov 03 23:00:39.648528 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.648368 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80"} err="failed to get container status \"1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80\": rpc error: code = NotFound desc = could not find container \"1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80\": container with ID starting with 1f46a4413dd2f054c244b8310895f33b6d6f0a37d2579225d07ff5f1a5bb1d80 not found: ID does not exist" Nov 03 23:00:39.648528 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.648390 4903 scope.go:122] "RemoveContainer" containerID="d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43" Nov 03 23:00:39.659220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.659201 4903 scope.go:122] "RemoveContainer" containerID="d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43" Nov 03 23:00:39.659652 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:39.659625 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43\": container with ID starting with d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43 not found: ID does not exist" containerID="d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43" Nov 03 23:00:39.659792 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.659660 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43"} err="failed to get container status \"d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43\": rpc error: code = NotFound desc = could not find container \"d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43\": container with ID starting with d34d7eb929627c5aa6f773d9f4aa2741d0049e4b23b288ebdbf5c945e3e60e43 not found: ID does not exist" Nov 03 23:00:39.684022 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.683985 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z"] Nov 03 23:00:39.688128 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.688101 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-68410c-68bc774c4c-z576z"] Nov 03 23:00:39.721048 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.721018 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:39.733440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.733390 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" podStartSLOduration=1.7333757840000001 podStartE2EDuration="1.733375784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:39.731933159 +0000 UTC m=+2866.135874136" watchObservedRunningTime="2026-11-03 23:00:39.733375784 +0000 UTC m=+2866.137316759" Nov 03 23:00:39.736913 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.736882 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05575897-8937-4040-97ea-3433a962f53b" path="/var/lib/kubelet/pods/05575897-8937-4040-97ea-3433a962f53b/volumes" Nov 03 23:00:39.737540 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.737520 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4644b46-9129-4f5e-982f-e187da494ff5" path="/var/lib/kubelet/pods/d4644b46-9129-4f5e-982f-e187da494ff5/volumes" Nov 03 23:00:39.767392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.767354 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr"] Nov 03 23:00:39.771558 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:39.771499 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-68410c-55c49569d6-rvsbr"] Nov 03 23:00:40.551380 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:40.551352 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:00:40.551541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:40.551420 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:00:41.736377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:41.736341 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147459de-4415-4d95-b2a8-c3de6e4ecc8e" path="/var/lib/kubelet/pods/147459de-4415-4d95-b2a8-c3de6e4ecc8e/volumes" Nov 03 23:00:42.158886 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.158848 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc"] Nov 03 23:00:42.159988 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.159957 4903 cpu_manager.go:414] "containerMap: removing container" podUID="05575897-8937-4040-97ea-3433a962f53b" containerName="payload-pre-processing" Nov 03 23:00:42.159988 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.159987 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05575897-8937-4040-97ea-3433a962f53b" containerName="payload-pre-processing" Nov 03 23:00:42.160155 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.160032 4903 cpu_manager.go:414] "containerMap: removing container" podUID="147459de-4415-4d95-b2a8-c3de6e4ecc8e" containerName="payload-processing" Nov 03 23:00:42.160155 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.160039 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147459de-4415-4d95-b2a8-c3de6e4ecc8e" containerName="payload-processing" Nov 03 23:00:42.160277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.160262 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="05575897-8937-4040-97ea-3433a962f53b" containerName="payload-pre-processing" Nov 03 23:00:42.160333 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.160307 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="147459de-4415-4d95-b2a8-c3de6e4ecc8e" containerName="payload-processing" Nov 03 23:00:42.165719 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.165693 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.200138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.198242 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc"] Nov 03 23:00:42.200763 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.200676 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-26f8ba\"/\"kube-root-ca.crt\"" Nov 03 23:00:42.200858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.200778 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-26f8ba\"/\"default-dockercfg-cml95\"" Nov 03 23:00:42.200858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.200696 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-26f8ba\"/\"rate-test-model-26f8ba-kserve-self-signed-certs\"" Nov 03 23:00:42.200858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.200829 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-26f8ba\"/\"openshift-service-ca.crt\"" Nov 03 23:00:42.233962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.233924 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-model-cache\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.234114 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.234095 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tmp-dir\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.234153 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.234143 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-home\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.234212 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.234184 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-dshm\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.234322 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.234304 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hln\" (UniqueName: \"kubernetes.io/projected/ffe7b53a-4da6-4dba-b7db-805b8673faf4-kube-api-access-s7hln\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.234367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.234347 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tls-certs\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.335051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.335017 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-home\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.335051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.335054 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-dshm\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.335329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.335164 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hln\" (UniqueName: \"kubernetes.io/projected/ffe7b53a-4da6-4dba-b7db-805b8673faf4-kube-api-access-s7hln\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.335329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.335216 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tls-certs\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.335329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.335280 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-model-cache\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.335518 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.335415 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tmp-dir\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.335598 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.335576 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-home\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.335716 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.335691 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-model-cache\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.335764 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.335728 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tmp-dir\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.337306 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.337276 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-dshm\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.337708 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.337691 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tls-certs\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.356399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.356371 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hln\" (UniqueName: \"kubernetes.io/projected/ffe7b53a-4da6-4dba-b7db-805b8673faf4-kube-api-access-s7hln\") pod \"rate-test-model-26f8ba-kserve-57bc466989-j8lbc\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.407621 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.407582 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql"] Nov 03 23:00:42.413093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.413029 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.426782 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.426755 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:00:42.430144 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.430123 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:00:42.430401 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.430373 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:42.430840 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.430475 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:00:42.431608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.431581 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql"] Nov 03 23:00:42.439191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.439071 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e5214e-k2vql\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.439492 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.439459 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e5214e-k2vql\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.439636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.439608 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rt8\" (UniqueName: \"kubernetes.io/projected/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-kube-api-access-67rt8\") pod \"maas-api-revoke-sub-a7e5214e-k2vql\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.507989 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.507954 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:42.541861 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.541824 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rt8\" (UniqueName: \"kubernetes.io/projected/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-kube-api-access-67rt8\") pod \"maas-api-revoke-sub-a7e5214e-k2vql\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.542069 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.542053 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e5214e-k2vql\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.542155 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.542137 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e5214e-k2vql\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.542977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.542952 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e5214e-k2vql\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.545953 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.545925 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e5214e-k2vql\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.565430 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.565398 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rt8\" (UniqueName: \"kubernetes.io/projected/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-kube-api-access-67rt8\") pod \"maas-api-revoke-sub-a7e5214e-k2vql\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.738207 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.736688 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:42.827948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:42.827916 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc"] Nov 03 23:00:42.832237 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:42.832200 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffe7b53a_4da6_4dba_b7db_805b8673faf4.slice/crio-e96392f54a1ad170fe7b6eb0cd2d61b2b250362ac5a757b730da2b1567c56e8a WatchSource:0}: Error finding container e96392f54a1ad170fe7b6eb0cd2d61b2b250362ac5a757b730da2b1567c56e8a: Status 404 returned error can't find the container with id e96392f54a1ad170fe7b6eb0cd2d61b2b250362ac5a757b730da2b1567c56e8a Nov 03 23:00:43.051597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:43.051569 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql"] Nov 03 23:00:43.051831 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:00:43.051805 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b2f6c2_fc47_4d96_9923_488ae8e4f286.slice/crio-c3ce4666266d8010a814da95bc7a63b9c0f3fec594ef4841a40081c63bc285c9 WatchSource:0}: Error finding container c3ce4666266d8010a814da95bc7a63b9c0f3fec594ef4841a40081c63bc285c9: Status 404 returned error can't find the container with id c3ce4666266d8010a814da95bc7a63b9c0f3fec594ef4841a40081c63bc285c9 Nov 03 23:00:43.694336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:43.694265 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" event={"ID":"ffe7b53a-4da6-4dba-b7db-805b8673faf4","Type":"ContainerStarted","Data":"d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa"} Nov 03 23:00:43.694583 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:43.694559 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" event={"ID":"ffe7b53a-4da6-4dba-b7db-805b8673faf4","Type":"ContainerStarted","Data":"e96392f54a1ad170fe7b6eb0cd2d61b2b250362ac5a757b730da2b1567c56e8a"} Nov 03 23:00:43.696423 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:43.696395 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" event={"ID":"a9b2f6c2-fc47-4d96-9923-488ae8e4f286","Type":"ContainerStarted","Data":"dbc7d996452f654fc0bab1992f85ca39a6df2e6f867e22c39c36ac46f1d50e02"} Nov 03 23:00:43.696549 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:43.696476 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" event={"ID":"a9b2f6c2-fc47-4d96-9923-488ae8e4f286","Type":"ContainerStarted","Data":"c3ce4666266d8010a814da95bc7a63b9c0f3fec594ef4841a40081c63bc285c9"} Nov 03 23:00:43.734175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:43.734123 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" podStartSLOduration=1.7341063220000001 podStartE2EDuration="1.734106322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:43.732855011 +0000 UTC m=+2870.136795987" watchObservedRunningTime="2026-11-03 23:00:43.734106322 +0000 UTC m=+2870.138047304" Nov 03 23:00:43.770661 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:43.770602 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" podStartSLOduration=1.770584585 podStartE2EDuration="1.770584585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:43.768721626 +0000 UTC m=+2870.172662798" watchObservedRunningTime="2026-11-03 23:00:43.770584585 +0000 UTC m=+2870.174525561" Nov 03 23:00:44.508282 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:44.508251 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:44.708426 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:44.708389 4903 generic.go:356] "Generic (PLEG): container finished" podID="a9b2f6c2-fc47-4d96-9923-488ae8e4f286" containerID="dbc7d996452f654fc0bab1992f85ca39a6df2e6f867e22c39c36ac46f1d50e02" exitCode=0 Nov 03 23:00:44.708638 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:44.708477 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" event={"ID":"a9b2f6c2-fc47-4d96-9923-488ae8e4f286","Type":"ContainerDied","Data":"dbc7d996452f654fc0bab1992f85ca39a6df2e6f867e22c39c36ac46f1d50e02"} Nov 03 23:00:45.915214 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:45.915181 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:45.990308 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:45.990223 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-kube-api-access-67rt8\" (UniqueName: \"kubernetes.io/projected/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-kube-api-access-67rt8\") pod \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " Nov 03 23:00:45.990458 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:45.990418 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-xks-ca\") pod \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " Nov 03 23:00:45.990509 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:45.990490 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-openshift-ca\") pod \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\" (UID: \"a9b2f6c2-fc47-4d96-9923-488ae8e4f286\") " Nov 03 23:00:45.990951 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:45.990916 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-openshift-ca" pod "a9b2f6c2-fc47-4d96-9923-488ae8e4f286" (UID: "a9b2f6c2-fc47-4d96-9923-488ae8e4f286"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:45.992581 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:45.992552 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-xks-ca" pod "a9b2f6c2-fc47-4d96-9923-488ae8e4f286" (UID: "a9b2f6c2-fc47-4d96-9923-488ae8e4f286"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:45.992748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:45.992726 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-kube-api-access-67rt8" pod "a9b2f6c2-fc47-4d96-9923-488ae8e4f286" (UID: "a9b2f6c2-fc47-4d96-9923-488ae8e4f286"). InnerVolumeSpecName "kube-api-access-67rt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:46.092056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:46.092025 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:46.092056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:46.092053 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:46.092056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:46.092064 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rt8\" (UniqueName: \"kubernetes.io/projected/a9b2f6c2-fc47-4d96-9923-488ae8e4f286-kube-api-access-67rt8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:46.809477 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:46.805758 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" event={"ID":"a9b2f6c2-fc47-4d96-9923-488ae8e4f286","Type":"ContainerDied","Data":"c3ce4666266d8010a814da95bc7a63b9c0f3fec594ef4841a40081c63bc285c9"} Nov 03 23:00:46.809477 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:46.805849 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3ce4666266d8010a814da95bc7a63b9c0f3fec594ef4841a40081c63bc285c9" Nov 03 23:00:46.809477 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:46.805961 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql" Nov 03 23:00:47.388829 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.388785 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9"] Nov 03 23:00:47.389654 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.389635 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a9b2f6c2-fc47-4d96-9923-488ae8e4f286" containerName="revoke-keys" Nov 03 23:00:47.389710 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.389655 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b2f6c2-fc47-4d96-9923-488ae8e4f286" containerName="revoke-keys" Nov 03 23:00:47.389800 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.389791 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b2f6c2-fc47-4d96-9923-488ae8e4f286" containerName="revoke-keys" Nov 03 23:00:47.393864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.393846 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.410029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.410005 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9"] Nov 03 23:00:47.414768 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.414741 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:00:47.414911 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.414776 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:00:47.428607 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.428580 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:00:47.428903 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.428884 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:00:47.432138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.432105 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8nj9\" (UniqueName: \"kubernetes.io/projected/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-kube-api-access-w8nj9\") pod \"maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.432344 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.432310 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.432432 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.432365 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.533438 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.533403 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8nj9\" (UniqueName: \"kubernetes.io/projected/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-kube-api-access-w8nj9\") pod \"maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.533643 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.533586 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.533643 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.533633 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.534560 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.534457 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.536650 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.536624 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.555019 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.554988 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8nj9\" (UniqueName: \"kubernetes.io/projected/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-kube-api-access-w8nj9\") pod \"maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:47.709477 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:47.709443 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:48.040534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:48.040503 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9"] Nov 03 23:00:48.386137 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:00:48.386048 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:00:48.721459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:48.721425 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:48.734216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:48.734191 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:48.827405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:48.827370 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" event={"ID":"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03","Type":"ContainerStarted","Data":"35cf692a37c8053d6a081a0667ad79fd1e2ec3c30cd3920d5ea00c016b55b149"} Nov 03 23:00:48.827586 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:48.827436 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" event={"ID":"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03","Type":"ContainerStarted","Data":"370c2e6a0f4f46829b9f01cffdca36d061f6f79dbd175607b2ee282ff99fa180"} Nov 03 23:00:48.839006 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:48.838981 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:00:48.863849 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:48.863806 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" podStartSLOduration=1.863793305 podStartE2EDuration="1.863793305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:00:48.863203606 +0000 UTC m=+2875.267144581" watchObservedRunningTime="2026-11-03 23:00:48.863793305 +0000 UTC m=+2875.267734279" Nov 03 23:00:49.837526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:49.837491 4903 generic.go:356] "Generic (PLEG): container finished" podID="56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03" containerID="35cf692a37c8053d6a081a0667ad79fd1e2ec3c30cd3920d5ea00c016b55b149" exitCode=0 Nov 03 23:00:49.838021 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:49.837580 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" event={"ID":"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03","Type":"ContainerDied","Data":"35cf692a37c8053d6a081a0667ad79fd1e2ec3c30cd3920d5ea00c016b55b149"} Nov 03 23:00:51.011794 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.011769 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:51.090330 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.090281 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-openshift-ca\") pod \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " Nov 03 23:00:51.090509 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.090351 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-xks-ca\") pod \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " Nov 03 23:00:51.090509 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.090413 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-kube-api-access-w8nj9\" (UniqueName: \"kubernetes.io/projected/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-kube-api-access-w8nj9\") pod \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\" (UID: \"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03\") " Nov 03 23:00:51.090715 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.090662 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-openshift-ca" pod "56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03" (UID: "56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:00:51.091041 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.091013 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:51.092551 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.092521 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-kube-api-access-w8nj9" pod "56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03" (UID: "56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03"). InnerVolumeSpecName "kube-api-access-w8nj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:00:51.092648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.092574 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-xks-ca" pod "56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03" (UID: "56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:00:51.192895 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.192860 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8nj9\" (UniqueName: \"kubernetes.io/projected/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-kube-api-access-w8nj9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:51.192895 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.192889 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:00:51.857081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.857051 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" Nov 03 23:00:51.857081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.857063 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9" event={"ID":"56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03","Type":"ContainerDied","Data":"370c2e6a0f4f46829b9f01cffdca36d061f6f79dbd175607b2ee282ff99fa180"} Nov 03 23:00:51.857373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:51.857105 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="370c2e6a0f4f46829b9f01cffdca36d061f6f79dbd175607b2ee282ff99fa180" Nov 03 23:00:52.508717 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:52.508682 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:52.521909 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:52.521886 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:52.877567 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:52.877480 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:00:59.632201 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:59.632164 4903 scope.go:122] "RemoveContainer" containerID="483e9c56ac371efdd86f32c7e41b21124c056e1d13dcc747cffc9c02c86218bd" Nov 03 23:00:59.643681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:59.643657 4903 scope.go:122] "RemoveContainer" containerID="ea844248517033d34b840a9177914116bcdb5b7fccec2179c13f2420380d47a1" Nov 03 23:00:59.654405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:59.654382 4903 scope.go:122] "RemoveContainer" containerID="eb5a53a399f0103f9366ff38df847596ff43c03913362628d8745cb96d44666b" Nov 03 23:00:59.666563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:59.666539 4903 scope.go:122] "RemoveContainer" containerID="94ba2dc2cc058162a3f3f2fc92a94e338abdb7481a717c8932afe17068e49736" Nov 03 23:00:59.677040 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:00:59.677019 4903 scope.go:122] "RemoveContainer" containerID="ec82397e7e647dd7115dd05ea7d0983b35bad79a2a9c4e98d62ab97730310280" Nov 03 23:01:01.336118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.336084 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s"] Nov 03 23:01:01.336655 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.336286 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" podUID="3549d146-0578-48ba-80f2-713aa4373d38" containerName="payload-processing" containerID="cri-o://ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0" gracePeriod=30 Nov 03 23:01:01.368695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.368663 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8"] Nov 03 23:01:01.368922 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.368892 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" podUID="a5dde97b-cbf4-4441-b5fc-c8ea75e58301" containerName="payload-pre-processing" containerID="cri-o://eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237" gracePeriod=30 Nov 03 23:01:01.737343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.737281 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 23:01:01.799384 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.798801 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 23:01:01.854234 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.854201 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-plugins-config-volume\") pod \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\" (UID: \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\") " Nov 03 23:01:01.854476 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.854412 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3549d146-0578-48ba-80f2-713aa4373d38-kube-api-access-88nvs\" (UniqueName: \"kubernetes.io/projected/3549d146-0578-48ba-80f2-713aa4373d38-kube-api-access-88nvs\") pod \"3549d146-0578-48ba-80f2-713aa4373d38\" (UID: \"3549d146-0578-48ba-80f2-713aa4373d38\") " Nov 03 23:01:01.854625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.854599 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3549d146-0578-48ba-80f2-713aa4373d38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549d146-0578-48ba-80f2-713aa4373d38-plugins-config-volume\") pod \"3549d146-0578-48ba-80f2-713aa4373d38\" (UID: \"3549d146-0578-48ba-80f2-713aa4373d38\") " Nov 03 23:01:01.855706 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.855680 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-plugins-config-volume" pod "a5dde97b-cbf4-4441-b5fc-c8ea75e58301" (UID: "a5dde97b-cbf4-4441-b5fc-c8ea75e58301"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:01.856049 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.856026 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3549d146-0578-48ba-80f2-713aa4373d38-plugins-config-volume" pod "3549d146-0578-48ba-80f2-713aa4373d38" (UID: "3549d146-0578-48ba-80f2-713aa4373d38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:01.857771 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.857747 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3549d146-0578-48ba-80f2-713aa4373d38-kube-api-access-88nvs" pod "3549d146-0578-48ba-80f2-713aa4373d38" (UID: "3549d146-0578-48ba-80f2-713aa4373d38"). InnerVolumeSpecName "kube-api-access-88nvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:01.947826 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.947789 4903 generic.go:356] "Generic (PLEG): container finished" podID="3549d146-0578-48ba-80f2-713aa4373d38" containerID="ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0" exitCode=0 Nov 03 23:01:01.948060 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.947844 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" Nov 03 23:01:01.948060 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.947877 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" event={"ID":"3549d146-0578-48ba-80f2-713aa4373d38","Type":"ContainerDied","Data":"ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0"} Nov 03 23:01:01.948060 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.947929 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s" event={"ID":"3549d146-0578-48ba-80f2-713aa4373d38","Type":"ContainerDied","Data":"926cb051f87cf07de1682208f1b67a3b470e8441ba8a339f827481068671a9cd"} Nov 03 23:01:01.948060 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.947965 4903 scope.go:122] "RemoveContainer" containerID="ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0" Nov 03 23:01:01.949666 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.949642 4903 generic.go:356] "Generic (PLEG): container finished" podID="a5dde97b-cbf4-4441-b5fc-c8ea75e58301" containerID="eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237" exitCode=0 Nov 03 23:01:01.949808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.949690 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" Nov 03 23:01:01.949808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.949726 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" event={"ID":"a5dde97b-cbf4-4441-b5fc-c8ea75e58301","Type":"ContainerDied","Data":"eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237"} Nov 03 23:01:01.949808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.949784 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8" event={"ID":"a5dde97b-cbf4-4441-b5fc-c8ea75e58301","Type":"ContainerDied","Data":"bdb72520aef5f082d6036d2d4f328c76ab7ea6acd310542366436fea88128ab1"} Nov 03 23:01:01.955701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.955678 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-kube-api-access-gxw96\" (UniqueName: \"kubernetes.io/projected/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-kube-api-access-gxw96\") pod \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\" (UID: \"a5dde97b-cbf4-4441-b5fc-c8ea75e58301\") " Nov 03 23:01:01.956399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.956378 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88nvs\" (UniqueName: \"kubernetes.io/projected/3549d146-0578-48ba-80f2-713aa4373d38-kube-api-access-88nvs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:01.956511 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.956431 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3549d146-0578-48ba-80f2-713aa4373d38-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:01.956511 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.956450 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:01.958617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.958591 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-kube-api-access-gxw96" pod "a5dde97b-cbf4-4441-b5fc-c8ea75e58301" (UID: "a5dde97b-cbf4-4441-b5fc-c8ea75e58301"). InnerVolumeSpecName "kube-api-access-gxw96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:01.960907 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.960874 4903 scope.go:122] "RemoveContainer" containerID="ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0" Nov 03 23:01:01.961289 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:01.961257 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0\": container with ID starting with ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0 not found: ID does not exist" containerID="ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0" Nov 03 23:01:01.961377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.961312 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0"} err="failed to get container status \"ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0\": rpc error: code = NotFound desc = could not find container \"ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0\": container with ID starting with ea538351e230cc1dc07d38ef5e0f4a6c167f1d68df85cd5eaa40dc2eecaf82c0 not found: ID does not exist" Nov 03 23:01:01.961377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.961336 4903 scope.go:122] "RemoveContainer" containerID="eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237" Nov 03 23:01:01.963557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.963537 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9"] Nov 03 23:01:01.968581 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.968556 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ff80c0-prvw9"] Nov 03 23:01:01.972093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.972071 4903 scope.go:122] "RemoveContainer" containerID="eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237" Nov 03 23:01:01.972416 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:01.972397 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237\": container with ID starting with eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237 not found: ID does not exist" containerID="eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237" Nov 03 23:01:01.972510 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:01.972426 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237"} err="failed to get container status \"eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237\": rpc error: code = NotFound desc = could not find container \"eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237\": container with ID starting with eb884367fe1e58b5dd1d00411c77262d978895a36557b3bca510ac9791c3c237 not found: ID does not exist" Nov 03 23:01:02.012262 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:02.012225 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s"] Nov 03 23:01:02.016938 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:02.016911 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ff80c0-55ddc664f8-xc54s"] Nov 03 23:01:02.059445 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:02.059401 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxw96\" (UniqueName: \"kubernetes.io/projected/a5dde97b-cbf4-4441-b5fc-c8ea75e58301-kube-api-access-gxw96\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:02.297093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:02.297055 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8"] Nov 03 23:01:02.301157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:02.301126 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ff80c0-5c7dbc8495-5pnw8"] Nov 03 23:01:03.737250 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:03.737210 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3549d146-0578-48ba-80f2-713aa4373d38" path="/var/lib/kubelet/pods/3549d146-0578-48ba-80f2-713aa4373d38/volumes" Nov 03 23:01:03.737929 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:03.737910 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03" path="/var/lib/kubelet/pods/56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03/volumes" Nov 03 23:01:03.738450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:03.738434 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5dde97b-cbf4-4441-b5fc-c8ea75e58301" path="/var/lib/kubelet/pods/a5dde97b-cbf4-4441-b5fc-c8ea75e58301/volumes" Nov 03 23:01:09.858887 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:09.858846 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6"] Nov 03 23:01:09.859483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:09.859106 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerName="istio-proxy" containerID="cri-o://506f4628e7c5e6219e9ee67d7d33edf555cf576189cad7d87cf516047d60ade8" gracePeriod=30 Nov 03 23:01:10.594208 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.594171 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw"] Nov 03 23:01:10.594920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.594906 4903 cpu_manager.go:414] "containerMap: removing container" podUID="56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03" containerName="revoke-keys" Nov 03 23:01:10.594975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.594922 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03" containerName="revoke-keys" Nov 03 23:01:10.594975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.594943 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a5dde97b-cbf4-4441-b5fc-c8ea75e58301" containerName="payload-pre-processing" Nov 03 23:01:10.594975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.594949 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5dde97b-cbf4-4441-b5fc-c8ea75e58301" containerName="payload-pre-processing" Nov 03 23:01:10.594975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.594964 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3549d146-0578-48ba-80f2-713aa4373d38" containerName="payload-processing" Nov 03 23:01:10.594975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.594969 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3549d146-0578-48ba-80f2-713aa4373d38" containerName="payload-processing" Nov 03 23:01:10.595156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.595130 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c20f36-6ed4-4cc7-8cb2-bd6b225a8f03" containerName="revoke-keys" Nov 03 23:01:10.595156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.595142 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5dde97b-cbf4-4441-b5fc-c8ea75e58301" containerName="payload-pre-processing" Nov 03 23:01:10.595156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.595150 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3549d146-0578-48ba-80f2-713aa4373d38" containerName="payload-processing" Nov 03 23:01:10.600762 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.600743 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.616964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.616938 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw"] Nov 03 23:01:10.621632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.621601 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:01:10.621632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.621604 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:01:10.657438 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.657403 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:01:10.657642 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.657497 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:01:10.684362 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.684326 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.684532 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.684475 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.684708 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.684690 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nccz\" (UniqueName: \"kubernetes.io/projected/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-kube-api-access-2nccz\") pod \"maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.785940 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.785894 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nccz\" (UniqueName: \"kubernetes.io/projected/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-kube-api-access-2nccz\") pod \"maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.786142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.785969 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.786142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.786097 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.786781 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.786753 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.788510 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.788489 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.807364 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.807334 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nccz\" (UniqueName: \"kubernetes.io/projected/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-kube-api-access-2nccz\") pod \"maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:10.915440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:10.915405 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:11.221339 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:11.221268 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw"] Nov 03 23:01:11.221556 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:11.221529 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0cd899_de8f_4ff4_bf9b_03fc5a132abe.slice/crio-4b82b45379e424e386864c05439aa3fe605d33d81982a62814ea9743c4d55101 WatchSource:0}: Error finding container 4b82b45379e424e386864c05439aa3fe605d33d81982a62814ea9743c4d55101: Status 404 returned error can't find the container with id 4b82b45379e424e386864c05439aa3fe605d33d81982a62814ea9743c4d55101 Nov 03 23:01:12.050181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:12.050139 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" event={"ID":"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe","Type":"ContainerStarted","Data":"d99fd53d1b038b7eae0440728e0ceba5c54f94aa75ac73f9d70b7656f07494c6"} Nov 03 23:01:12.050641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:12.050198 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" event={"ID":"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe","Type":"ContainerStarted","Data":"4b82b45379e424e386864c05439aa3fe605d33d81982a62814ea9743c4d55101"} Nov 03 23:01:12.094410 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:12.094348 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" podStartSLOduration=2.094330275 podStartE2EDuration="2.094330275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:12.091843307 +0000 UTC m=+2898.495784282" watchObservedRunningTime="2026-11-03 23:01:12.094330275 +0000 UTC m=+2898.498271251" Nov 03 23:01:13.059559 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:13.059522 4903 generic.go:356] "Generic (PLEG): container finished" podID="4a0cd899-de8f-4ff4-bf9b-03fc5a132abe" containerID="d99fd53d1b038b7eae0440728e0ceba5c54f94aa75ac73f9d70b7656f07494c6" exitCode=0 Nov 03 23:01:13.059964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:13.059574 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" event={"ID":"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe","Type":"ContainerDied","Data":"d99fd53d1b038b7eae0440728e0ceba5c54f94aa75ac73f9d70b7656f07494c6"} Nov 03 23:01:14.241764 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.241739 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:14.348268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.348196 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-openshift-ca\") pod \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " Nov 03 23:01:14.348439 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.348411 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-xks-ca\") pod \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " Nov 03 23:01:14.348484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.348449 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-kube-api-access-2nccz\" (UniqueName: \"kubernetes.io/projected/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-kube-api-access-2nccz\") pod \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\" (UID: \"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe\") " Nov 03 23:01:14.348583 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.348559 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-openshift-ca" pod "4a0cd899-de8f-4ff4-bf9b-03fc5a132abe" (UID: "4a0cd899-de8f-4ff4-bf9b-03fc5a132abe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:14.348880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.348865 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:14.350578 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.350555 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-xks-ca" pod "4a0cd899-de8f-4ff4-bf9b-03fc5a132abe" (UID: "4a0cd899-de8f-4ff4-bf9b-03fc5a132abe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:14.350685 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.350598 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-kube-api-access-2nccz" pod "4a0cd899-de8f-4ff4-bf9b-03fc5a132abe" (UID: "4a0cd899-de8f-4ff4-bf9b-03fc5a132abe"). InnerVolumeSpecName "kube-api-access-2nccz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:14.449734 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.449704 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:14.449734 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.449729 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nccz\" (UniqueName: \"kubernetes.io/projected/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe-kube-api-access-2nccz\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:14.807113 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:14.807077 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:01:15.086360 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.083472 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" Nov 03 23:01:15.086360 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.083471 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw" event={"ID":"4a0cd899-de8f-4ff4-bf9b-03fc5a132abe","Type":"ContainerDied","Data":"4b82b45379e424e386864c05439aa3fe605d33d81982a62814ea9743c4d55101"} Nov 03 23:01:15.086360 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.083608 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b82b45379e424e386864c05439aa3fe605d33d81982a62814ea9743c4d55101" Nov 03 23:01:15.086360 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.085914 4903 generic.go:356] "Generic (PLEG): container finished" podID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerID="506f4628e7c5e6219e9ee67d7d33edf555cf576189cad7d87cf516047d60ade8" exitCode=0 Nov 03 23:01:15.086360 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.085987 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" event={"ID":"7f02f88d-4e82-4bf0-adea-351a83a49ca5","Type":"ContainerDied","Data":"506f4628e7c5e6219e9ee67d7d33edf555cf576189cad7d87cf516047d60ade8"} Nov 03 23:01:15.162917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.162888 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 23:01:15.265439 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.265400 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-envoy\") pod \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " Nov 03 23:01:15.265971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.265535 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istiod-ca-cert\") pod \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " Nov 03 23:01:15.265971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.265593 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-credential-socket\") pod \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " Nov 03 23:01:15.265971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.265630 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-token\") pod \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " Nov 03 23:01:15.265971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.265658 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-kube-api-access-26xxm\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-kube-api-access-26xxm\") pod \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " Nov 03 23:01:15.265971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.265699 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-certs\") pod \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " Nov 03 23:01:15.265971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.265819 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-podinfo\") pod \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " Nov 03 23:01:15.265971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.265965 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-data\") pod \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " Nov 03 23:01:15.266425 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.266084 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-socket\") pod \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\" (UID: \"7f02f88d-4e82-4bf0-adea-351a83a49ca5\") " Nov 03 23:01:15.267322 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.267284 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-credential-socket" pod "7f02f88d-4e82-4bf0-adea-351a83a49ca5" (UID: "7f02f88d-4e82-4bf0-adea-351a83a49ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.267449 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.267315 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-socket" pod "7f02f88d-4e82-4bf0-adea-351a83a49ca5" (UID: "7f02f88d-4e82-4bf0-adea-351a83a49ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.267449 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.267404 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istiod-ca-cert" pod "7f02f88d-4e82-4bf0-adea-351a83a49ca5" (UID: "7f02f88d-4e82-4bf0-adea-351a83a49ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:15.267714 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.267666 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-data" pod "7f02f88d-4e82-4bf0-adea-351a83a49ca5" (UID: "7f02f88d-4e82-4bf0-adea-351a83a49ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.267714 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.267691 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-certs" pod "7f02f88d-4e82-4bf0-adea-351a83a49ca5" (UID: "7f02f88d-4e82-4bf0-adea-351a83a49ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.267877 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.267860 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-envoy" pod "7f02f88d-4e82-4bf0-adea-351a83a49ca5" (UID: "7f02f88d-4e82-4bf0-adea-351a83a49ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:15.269178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.269161 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-token" pod "7f02f88d-4e82-4bf0-adea-351a83a49ca5" (UID: "7f02f88d-4e82-4bf0-adea-351a83a49ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:15.269331 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.269309 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-podinfo" pod "7f02f88d-4e82-4bf0-adea-351a83a49ca5" (UID: "7f02f88d-4e82-4bf0-adea-351a83a49ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:01:15.269646 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.269629 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-kube-api-access-26xxm" pod "7f02f88d-4e82-4bf0-adea-351a83a49ca5" (UID: "7f02f88d-4e82-4bf0-adea-351a83a49ca5"). InnerVolumeSpecName "kube-api-access-26xxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:15.323102 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.323043 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh"] Nov 03 23:01:15.329860 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.329829 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk"] Nov 03 23:01:15.337058 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.336984 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm"] Nov 03 23:01:15.367908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.367875 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.367908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.367901 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.367908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.367915 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.368221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.367930 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.368221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.367939 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.368221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.367949 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.368221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.367957 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.368221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.367965 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26xxm\" (UniqueName: \"kubernetes.io/projected/7f02f88d-4e82-4bf0-adea-351a83a49ca5-kube-api-access-26xxm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:15.368221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:15.367974 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f02f88d-4e82-4bf0-adea-351a83a49ca5-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:16.118402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:16.111376 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" Nov 03 23:01:16.118402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:16.112144 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6" event={"ID":"7f02f88d-4e82-4bf0-adea-351a83a49ca5","Type":"ContainerDied","Data":"4ad882636a9ad374956bfcb4b564947ed3638df7d7897d00b983b047a3584650"} Nov 03 23:01:16.118402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:16.112203 4903 scope.go:122] "RemoveContainer" containerID="506f4628e7c5e6219e9ee67d7d33edf555cf576189cad7d87cf516047d60ade8" Nov 03 23:01:16.173388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:16.170711 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6"] Nov 03 23:01:16.179557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:16.179517 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ff80c0-openshift-default-6587d8df9f-nbfj6"] Nov 03 23:01:17.737331 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:17.737272 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" path="/var/lib/kubelet/pods/7f02f88d-4e82-4bf0-adea-351a83a49ca5/volumes" Nov 03 23:01:18.385786 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:18.385753 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:01:22.119687 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:22.119655 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44"] Nov 03 23:01:25.798340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:25.798311 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:01:25.994271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:25.994238 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:01:26.981785 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:26.981748 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx"] Nov 03 23:01:26.982517 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:26.982501 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4a0cd899-de8f-4ff4-bf9b-03fc5a132abe" containerName="revoke-keys" Nov 03 23:01:26.982571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:26.982518 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0cd899-de8f-4ff4-bf9b-03fc5a132abe" containerName="revoke-keys" Nov 03 23:01:26.982571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:26.982530 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerName="istio-proxy" Nov 03 23:01:26.982571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:26.982535 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerName="istio-proxy" Nov 03 23:01:26.982678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:26.982651 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0cd899-de8f-4ff4-bf9b-03fc5a132abe" containerName="revoke-keys" Nov 03 23:01:26.982678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:26.982659 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f02f88d-4e82-4bf0-adea-351a83a49ca5" containerName="istio-proxy" Nov 03 23:01:26.986454 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:26.986435 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.005896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.005867 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx"] Nov 03 23:01:27.010408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.010385 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:01:27.010408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.010386 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:01:27.010408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.010388 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:01:27.010655 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.010441 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:01:27.040485 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.040442 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.040636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.040555 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxd2g\" (UniqueName: \"kubernetes.io/projected/0d5d0ae5-dca1-4104-8379-92e90cd60201-kube-api-access-sxd2g\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.040636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.040601 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.120175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.120138 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" podUID="6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7" containerName="payload-processing" containerID="cri-o://56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d" gracePeriod=25 Nov 03 23:01:27.141537 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.141504 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.141732 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.141717 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.141786 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.141769 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxd2g\" (UniqueName: \"kubernetes.io/projected/0d5d0ae5-dca1-4104-8379-92e90cd60201-kube-api-access-sxd2g\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.142186 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.142165 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.143998 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.143981 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.163695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.163668 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxd2g\" (UniqueName: \"kubernetes.io/projected/0d5d0ae5-dca1-4104-8379-92e90cd60201-kube-api-access-sxd2g\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.301476 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.301445 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:27.460346 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.458446 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 23:01:27.551497 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.551405 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-plugins-config-volume\") pod \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\" (UID: \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\") " Nov 03 23:01:27.551678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.551625 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-kube-api-access-zfgd6\" (UniqueName: \"kubernetes.io/projected/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-kube-api-access-zfgd6\") pod \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\" (UID: \"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7\") " Nov 03 23:01:27.551874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.551848 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-plugins-config-volume" pod "6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7" (UID: "6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:27.552344 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.552322 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:27.554173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.554149 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-kube-api-access-zfgd6" pod "6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7" (UID: "6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7"). InnerVolumeSpecName "kube-api-access-zfgd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:27.625640 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.625611 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx"] Nov 03 23:01:27.627114 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:27.627088 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5d0ae5_dca1_4104_8379_92e90cd60201.slice/crio-796ab569d7a02608e6d3987999cb1655314a50e64c043b9e38182442ab75befd WatchSource:0}: Error finding container 796ab569d7a02608e6d3987999cb1655314a50e64c043b9e38182442ab75befd: Status 404 returned error can't find the container with id 796ab569d7a02608e6d3987999cb1655314a50e64c043b9e38182442ab75befd Nov 03 23:01:27.653390 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:27.653365 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfgd6\" (UniqueName: \"kubernetes.io/projected/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7-kube-api-access-zfgd6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:28.217457 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.217363 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" event={"ID":"0d5d0ae5-dca1-4104-8379-92e90cd60201","Type":"ContainerStarted","Data":"4945c03ff3c70623472fdf81a58be294307ed2f00de4464b0c31ae6fe3401c69"} Nov 03 23:01:28.217457 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.217426 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" event={"ID":"0d5d0ae5-dca1-4104-8379-92e90cd60201","Type":"ContainerStarted","Data":"796ab569d7a02608e6d3987999cb1655314a50e64c043b9e38182442ab75befd"} Nov 03 23:01:28.218798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.218768 4903 generic.go:356] "Generic (PLEG): container finished" podID="6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7" containerID="56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d" exitCode=0 Nov 03 23:01:28.218932 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.218801 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" Nov 03 23:01:28.218932 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.218860 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" event={"ID":"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7","Type":"ContainerDied","Data":"56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d"} Nov 03 23:01:28.218932 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.218906 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44" event={"ID":"6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7","Type":"ContainerDied","Data":"a5a616f7d09a101591dc56843e22b2256f502a1d154b907df432bcb75b277806"} Nov 03 23:01:28.218932 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.218928 4903 scope.go:122] "RemoveContainer" containerID="56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d" Nov 03 23:01:28.229556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.229536 4903 scope.go:122] "RemoveContainer" containerID="56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d" Nov 03 23:01:28.229825 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:28.229802 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d\": container with ID starting with 56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d not found: ID does not exist" containerID="56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d" Nov 03 23:01:28.229896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.229834 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d"} err="failed to get container status \"56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d\": rpc error: code = NotFound desc = could not find container \"56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d\": container with ID starting with 56c69ee2931f9b574dcb03a7e9762137fd30f2664cb0ddcde7c623a79021c48d not found: ID does not exist" Nov 03 23:01:28.262865 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.262812 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" podStartSLOduration=2.262798771 podStartE2EDuration="2.262798771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:28.259748173 +0000 UTC m=+2914.663689150" watchObservedRunningTime="2026-11-03 23:01:28.262798771 +0000 UTC m=+2914.666739745" Nov 03 23:01:28.302046 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.302016 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44"] Nov 03 23:01:28.310677 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:28.310643 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e19850-565c99cd8d-qhx44"] Nov 03 23:01:29.228131 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:29.228098 4903 generic.go:356] "Generic (PLEG): container finished" podID="0d5d0ae5-dca1-4104-8379-92e90cd60201" containerID="4945c03ff3c70623472fdf81a58be294307ed2f00de4464b0c31ae6fe3401c69" exitCode=0 Nov 03 23:01:29.228631 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:29.228180 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" event={"ID":"0d5d0ae5-dca1-4104-8379-92e90cd60201","Type":"ContainerDied","Data":"4945c03ff3c70623472fdf81a58be294307ed2f00de4464b0c31ae6fe3401c69"} Nov 03 23:01:29.748135 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:29.748081 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7" path="/var/lib/kubelet/pods/6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7/volumes" Nov 03 23:01:30.350426 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.350375 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" containerID="cri-o://905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153" gracePeriod=45 Nov 03 23:01:30.357539 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.357503 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" containerID="cri-o://47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2" gracePeriod=45 Nov 03 23:01:30.359486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.359451 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" containerID="cri-o://5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d" gracePeriod=45 Nov 03 23:01:30.556792 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.555312 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:30.621750 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.620751 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-xks-ca\") pod \"0d5d0ae5-dca1-4104-8379-92e90cd60201\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " Nov 03 23:01:30.621750 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.620899 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-openshift-ca\") pod \"0d5d0ae5-dca1-4104-8379-92e90cd60201\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " Nov 03 23:01:30.621750 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.620994 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5d0ae5-dca1-4104-8379-92e90cd60201-kube-api-access-sxd2g\" (UniqueName: \"kubernetes.io/projected/0d5d0ae5-dca1-4104-8379-92e90cd60201-kube-api-access-sxd2g\") pod \"0d5d0ae5-dca1-4104-8379-92e90cd60201\" (UID: \"0d5d0ae5-dca1-4104-8379-92e90cd60201\") " Nov 03 23:01:30.622768 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.622736 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-openshift-ca" pod "0d5d0ae5-dca1-4104-8379-92e90cd60201" (UID: "0d5d0ae5-dca1-4104-8379-92e90cd60201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:30.628804 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.628769 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5d0ae5-dca1-4104-8379-92e90cd60201-kube-api-access-sxd2g" pod "0d5d0ae5-dca1-4104-8379-92e90cd60201" (UID: "0d5d0ae5-dca1-4104-8379-92e90cd60201"). InnerVolumeSpecName "kube-api-access-sxd2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:30.639816 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.639780 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-xks-ca" pod "0d5d0ae5-dca1-4104-8379-92e90cd60201" (UID: "0d5d0ae5-dca1-4104-8379-92e90cd60201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:30.659743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.659343 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 23:01:30.692911 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.692871 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 23:01:30.709216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.708878 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 23:01:30.723083 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.723047 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-model-cache\") pod \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " Nov 03 23:01:30.723330 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.723285 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-home\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-home\") pod \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " Nov 03 23:01:30.723450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.723360 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tmp-dir\") pod \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " Nov 03 23:01:30.723450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.723371 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-model-cache" pod "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" (UID: "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.723450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.723421 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-dshm\") pod \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " Nov 03 23:01:30.723653 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.723494 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-kube-api-access-5gg9r\" (UniqueName: \"kubernetes.io/projected/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-kube-api-access-5gg9r\") pod \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " Nov 03 23:01:30.723653 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.723524 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tls-certs\") pod \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\" (UID: \"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef\") " Nov 03 23:01:30.723653 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.723583 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-home" pod "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" (UID: "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.724117 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.724080 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tmp-dir" pod "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" (UID: "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.724225 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.724203 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.724225 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.724222 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.724351 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.724236 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5d0ae5-dca1-4104-8379-92e90cd60201-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.724351 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.724251 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxd2g\" (UniqueName: \"kubernetes.io/projected/0d5d0ae5-dca1-4104-8379-92e90cd60201-kube-api-access-sxd2g\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.724351 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.724265 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.724351 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.724322 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.726179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.726157 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-dshm" pod "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" (UID: "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.726473 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.726450 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-kube-api-access-5gg9r" pod "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" (UID: "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef"). InnerVolumeSpecName "kube-api-access-5gg9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:30.726582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.726550 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tls-certs" pod "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" (UID: "fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:30.824937 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.824863 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2b69ca-544e-4521-97a2-09a6cecf868a-kube-api-access-4p742\" (UniqueName: \"kubernetes.io/projected/2d2b69ca-544e-4521-97a2-09a6cecf868a-kube-api-access-4p742\") pod \"2d2b69ca-544e-4521-97a2-09a6cecf868a\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " Nov 03 23:01:30.824937 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.824904 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-home\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-home\") pod \"2d2b69ca-544e-4521-97a2-09a6cecf868a\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " Nov 03 23:01:30.825194 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.824967 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-dshm\") pod \"85dc74e3-303e-4920-9d9f-4e417621cb7a\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " Nov 03 23:01:30.825194 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825006 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-home\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-home\") pod \"85dc74e3-303e-4920-9d9f-4e417621cb7a\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " Nov 03 23:01:30.825194 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825060 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-tmp-dir\") pod \"2d2b69ca-544e-4521-97a2-09a6cecf868a\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " Nov 03 23:01:30.825194 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825170 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-dshm\") pod \"2d2b69ca-544e-4521-97a2-09a6cecf868a\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " Nov 03 23:01:30.825459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825205 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85dc74e3-303e-4920-9d9f-4e417621cb7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/85dc74e3-303e-4920-9d9f-4e417621cb7a-tls-certs\") pod \"85dc74e3-303e-4920-9d9f-4e417621cb7a\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " Nov 03 23:01:30.825459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825244 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-model-cache\") pod \"2d2b69ca-544e-4521-97a2-09a6cecf868a\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " Nov 03 23:01:30.825459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825265 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-home" pod "2d2b69ca-544e-4521-97a2-09a6cecf868a" (UID: "2d2b69ca-544e-4521-97a2-09a6cecf868a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.825459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825288 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-model-cache\") pod \"85dc74e3-303e-4920-9d9f-4e417621cb7a\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " Nov 03 23:01:30.825459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825381 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85dc74e3-303e-4920-9d9f-4e417621cb7a-kube-api-access-2q2cs\" (UniqueName: \"kubernetes.io/projected/85dc74e3-303e-4920-9d9f-4e417621cb7a-kube-api-access-2q2cs\") pod \"85dc74e3-303e-4920-9d9f-4e417621cb7a\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " Nov 03 23:01:30.825459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825418 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2b69ca-544e-4521-97a2-09a6cecf868a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2b69ca-544e-4521-97a2-09a6cecf868a-tls-certs\") pod \"2d2b69ca-544e-4521-97a2-09a6cecf868a\" (UID: \"2d2b69ca-544e-4521-97a2-09a6cecf868a\") " Nov 03 23:01:30.825459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825428 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-tmp-dir" pod "2d2b69ca-544e-4521-97a2-09a6cecf868a" (UID: "2d2b69ca-544e-4521-97a2-09a6cecf868a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.825459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825449 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-tmp-dir\") pod \"85dc74e3-303e-4920-9d9f-4e417621cb7a\" (UID: \"85dc74e3-303e-4920-9d9f-4e417621cb7a\") " Nov 03 23:01:30.825946 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.825861 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-model-cache" pod "2d2b69ca-544e-4521-97a2-09a6cecf868a" (UID: "2d2b69ca-544e-4521-97a2-09a6cecf868a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.826062 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.826042 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-tmp-dir" pod "85dc74e3-303e-4920-9d9f-4e417621cb7a" (UID: "85dc74e3-303e-4920-9d9f-4e417621cb7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.826281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.826262 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.826373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.826285 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.826373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.826317 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.826373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.826328 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gg9r\" (UniqueName: \"kubernetes.io/projected/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-kube-api-access-5gg9r\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.826373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.826343 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.826373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.826357 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.826373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.826368 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.826929 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.826877 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-home" pod "85dc74e3-303e-4920-9d9f-4e417621cb7a" (UID: "85dc74e3-303e-4920-9d9f-4e417621cb7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.827338 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.827082 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-model-cache" pod "85dc74e3-303e-4920-9d9f-4e417621cb7a" (UID: "85dc74e3-303e-4920-9d9f-4e417621cb7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.827866 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.827839 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-dshm" pod "85dc74e3-303e-4920-9d9f-4e417621cb7a" (UID: "85dc74e3-303e-4920-9d9f-4e417621cb7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.827978 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.827953 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-dshm" pod "2d2b69ca-544e-4521-97a2-09a6cecf868a" (UID: "2d2b69ca-544e-4521-97a2-09a6cecf868a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:01:30.828093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.828068 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2b69ca-544e-4521-97a2-09a6cecf868a-kube-api-access-4p742" pod "2d2b69ca-544e-4521-97a2-09a6cecf868a" (UID: "2d2b69ca-544e-4521-97a2-09a6cecf868a"). InnerVolumeSpecName "kube-api-access-4p742". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:30.828316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.828269 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85dc74e3-303e-4920-9d9f-4e417621cb7a-tls-certs" pod "85dc74e3-303e-4920-9d9f-4e417621cb7a" (UID: "85dc74e3-303e-4920-9d9f-4e417621cb7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:30.829137 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.829114 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2b69ca-544e-4521-97a2-09a6cecf868a-tls-certs" pod "2d2b69ca-544e-4521-97a2-09a6cecf868a" (UID: "2d2b69ca-544e-4521-97a2-09a6cecf868a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:30.829221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.829152 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dc74e3-303e-4920-9d9f-4e417621cb7a-kube-api-access-2q2cs" pod "85dc74e3-303e-4920-9d9f-4e417621cb7a" (UID: "85dc74e3-303e-4920-9d9f-4e417621cb7a"). InnerVolumeSpecName "kube-api-access-2q2cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:30.928084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.928030 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2b69ca-544e-4521-97a2-09a6cecf868a-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.928084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.928066 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dc74e3-303e-4920-9d9f-4e417621cb7a-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.928084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.928081 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.928368 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.928094 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q2cs\" (UniqueName: \"kubernetes.io/projected/85dc74e3-303e-4920-9d9f-4e417621cb7a-kube-api-access-2q2cs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.928368 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.928107 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2b69ca-544e-4521-97a2-09a6cecf868a-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.928368 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.928121 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p742\" (UniqueName: \"kubernetes.io/projected/2d2b69ca-544e-4521-97a2-09a6cecf868a-kube-api-access-4p742\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.928368 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.928132 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:30.928368 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:30.928142 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85dc74e3-303e-4920-9d9f-4e417621cb7a-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:31.246738 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.246701 4903 generic.go:356] "Generic (PLEG): container finished" podID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerID="5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d" exitCode=0 Nov 03 23:01:31.246936 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.246751 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" Nov 03 23:01:31.246936 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.246798 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" event={"ID":"85dc74e3-303e-4920-9d9f-4e417621cb7a","Type":"ContainerDied","Data":"5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d"} Nov 03 23:01:31.246936 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.246853 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm" event={"ID":"85dc74e3-303e-4920-9d9f-4e417621cb7a","Type":"ContainerDied","Data":"d12f02c8bca84e954d2d3123e9d2a8d6b2b0b54054e80775599fa993d2919591"} Nov 03 23:01:31.246936 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.246880 4903 scope.go:122] "RemoveContainer" containerID="5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d" Nov 03 23:01:31.248340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.248305 4903 generic.go:356] "Generic (PLEG): container finished" podID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerID="905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153" exitCode=0 Nov 03 23:01:31.248412 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.248367 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" Nov 03 23:01:31.248461 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.248435 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" event={"ID":"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef","Type":"ContainerDied","Data":"905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153"} Nov 03 23:01:31.248521 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.248472 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh" event={"ID":"fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef","Type":"ContainerDied","Data":"325846d298215a26688444b8b3493a037b24b79d9f74ad4c012d08b56b895342"} Nov 03 23:01:31.250167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.250138 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" event={"ID":"0d5d0ae5-dca1-4104-8379-92e90cd60201","Type":"ContainerDied","Data":"796ab569d7a02608e6d3987999cb1655314a50e64c043b9e38182442ab75befd"} Nov 03 23:01:31.250334 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.250184 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="796ab569d7a02608e6d3987999cb1655314a50e64c043b9e38182442ab75befd" Nov 03 23:01:31.250334 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.250195 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx" Nov 03 23:01:31.251699 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.251677 4903 generic.go:356] "Generic (PLEG): container finished" podID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerID="47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2" exitCode=0 Nov 03 23:01:31.251810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.251720 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" event={"ID":"2d2b69ca-544e-4521-97a2-09a6cecf868a","Type":"ContainerDied","Data":"47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2"} Nov 03 23:01:31.251810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.251744 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" event={"ID":"2d2b69ca-544e-4521-97a2-09a6cecf868a","Type":"ContainerDied","Data":"f39785c9edb40171438d78a9abd576439580ad9583d4ab0f2f7ac7406ac82b1c"} Nov 03 23:01:31.251810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.251800 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" Nov 03 23:01:31.260888 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.260869 4903 scope.go:122] "RemoveContainer" containerID="5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d" Nov 03 23:01:31.261210 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:31.261190 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d\": container with ID starting with 5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d not found: ID does not exist" containerID="5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d" Nov 03 23:01:31.261273 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.261219 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d"} err="failed to get container status \"5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d\": rpc error: code = NotFound desc = could not find container \"5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d\": container with ID starting with 5c39a6b58b29c9f71ed512d7540fdd86b1dea1c1dd8828cb1ce881f003926e4d not found: ID does not exist" Nov 03 23:01:31.261273 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.261242 4903 scope.go:122] "RemoveContainer" containerID="905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153" Nov 03 23:01:31.274608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.274587 4903 scope.go:122] "RemoveContainer" containerID="905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153" Nov 03 23:01:31.274962 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:31.274938 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153\": container with ID starting with 905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153 not found: ID does not exist" containerID="905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153" Nov 03 23:01:31.275058 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.274971 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153"} err="failed to get container status \"905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153\": rpc error: code = NotFound desc = could not find container \"905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153\": container with ID starting with 905e3021d8a3737dc6175f0f630ef11856a81be0cd925c56069a0ed6b26d7153 not found: ID does not exist" Nov 03 23:01:31.275058 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.274993 4903 scope.go:122] "RemoveContainer" containerID="47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2" Nov 03 23:01:31.286072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.286049 4903 scope.go:122] "RemoveContainer" containerID="47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2" Nov 03 23:01:31.286436 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:31.286415 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2\": container with ID starting with 47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2 not found: ID does not exist" containerID="47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2" Nov 03 23:01:31.286540 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.286443 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2"} err="failed to get container status \"47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2\": rpc error: code = NotFound desc = could not find container \"47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2\": container with ID starting with 47957700a5fe7d65789815d43354d63a5edf1b251bce46663deb46b62c4819d2 not found: ID does not exist" Nov 03 23:01:31.305028 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.305005 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm"] Nov 03 23:01:31.311220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.311198 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/simulator-w1-e19850-kserve-7c89799dcd-rnzhm"] Nov 03 23:01:31.380450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.380426 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh"] Nov 03 23:01:31.388510 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.388485 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/distinct-w1-e19850-kserve-c6f6cc95d-4r6dh"] Nov 03 23:01:31.419793 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.419762 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk"] Nov 03 23:01:31.426191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.426131 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk"] Nov 03 23:01:31.623009 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.622912 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e19850/premium-w1-e19850-kserve-59bcc8c6d7-xzcnk" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:01:31.738954 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.738916 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" path="/var/lib/kubelet/pods/2d2b69ca-544e-4521-97a2-09a6cecf868a/volumes" Nov 03 23:01:31.739559 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.739540 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" path="/var/lib/kubelet/pods/85dc74e3-303e-4920-9d9f-4e417621cb7a/volumes" Nov 03 23:01:31.740097 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:31.740081 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" path="/var/lib/kubelet/pods/fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef/volumes" Nov 03 23:01:35.608817 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.608788 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs"] Nov 03 23:01:35.609520 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609503 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7" containerName="payload-processing" Nov 03 23:01:35.609573 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609522 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7" containerName="payload-processing" Nov 03 23:01:35.609573 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609537 4903 cpu_manager.go:414] "containerMap: removing container" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" Nov 03 23:01:35.609573 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609542 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" Nov 03 23:01:35.609573 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609566 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0d5d0ae5-dca1-4104-8379-92e90cd60201" containerName="revoke-keys" Nov 03 23:01:35.609573 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609572 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5d0ae5-dca1-4104-8379-92e90cd60201" containerName="revoke-keys" Nov 03 23:01:35.609748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609609 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" Nov 03 23:01:35.609748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609617 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" Nov 03 23:01:35.609748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609632 4903 cpu_manager.go:414] "containerMap: removing container" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" Nov 03 23:01:35.609748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609636 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" Nov 03 23:01:35.609748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609737 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2b69ca-544e-4521-97a2-09a6cecf868a" containerName="main" Nov 03 23:01:35.609748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609749 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="85dc74e3-303e-4920-9d9f-4e417621cb7a" containerName="main" Nov 03 23:01:35.609941 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609757 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1f1640-d40b-4e0f-ae1f-2e1d53b5a4f7" containerName="payload-processing" Nov 03 23:01:35.609941 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609764 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd8a10b-f10c-4d22-8d3d-d3ed78acbdef" containerName="main" Nov 03 23:01:35.609941 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.609773 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5d0ae5-dca1-4104-8379-92e90cd60201" containerName="revoke-keys" Nov 03 23:01:35.615884 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.615839 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:35.629628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.629576 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ad693\"" Nov 03 23:01:35.631792 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.631769 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs"] Nov 03 23:01:35.637574 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.637548 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ad693-dockercfg-z2ndq\"" Nov 03 23:01:35.687697 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.687661 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7"] Nov 03 23:01:35.692265 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.692247 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:35.710416 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.710389 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7"] Nov 03 23:01:35.795167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.795124 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs\" (UID: \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:35.795415 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.795244 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsnm\" (UniqueName: \"kubernetes.io/projected/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-kube-api-access-knsnm\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs\" (UID: \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:35.795521 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.795501 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad1242d-05c2-4478-84fb-d3fe522001a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7\" (UID: \"aad1242d-05c2-4478-84fb-d3fe522001a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:35.795590 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.795561 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h474\" (UniqueName: \"kubernetes.io/projected/aad1242d-05c2-4478-84fb-d3fe522001a3-kube-api-access-8h474\") pod \"payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7\" (UID: \"aad1242d-05c2-4478-84fb-d3fe522001a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:35.896434 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.896354 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h474\" (UniqueName: \"kubernetes.io/projected/aad1242d-05c2-4478-84fb-d3fe522001a3-kube-api-access-8h474\") pod \"payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7\" (UID: \"aad1242d-05c2-4478-84fb-d3fe522001a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:35.896434 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.896393 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs\" (UID: \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:35.896633 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.896495 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsnm\" (UniqueName: \"kubernetes.io/projected/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-kube-api-access-knsnm\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs\" (UID: \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:35.896815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.896787 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad1242d-05c2-4478-84fb-d3fe522001a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7\" (UID: \"aad1242d-05c2-4478-84fb-d3fe522001a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:35.897592 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.897570 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad1242d-05c2-4478-84fb-d3fe522001a3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7\" (UID: \"aad1242d-05c2-4478-84fb-d3fe522001a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:35.897704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.897657 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs\" (UID: \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:35.903809 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.903784 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5"] Nov 03 23:01:35.918398 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.918368 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h474\" (UniqueName: \"kubernetes.io/projected/aad1242d-05c2-4478-84fb-d3fe522001a3-kube-api-access-8h474\") pod \"payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7\" (UID: \"aad1242d-05c2-4478-84fb-d3fe522001a3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:35.918519 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.918452 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsnm\" (UniqueName: \"kubernetes.io/projected/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-kube-api-access-knsnm\") pod \"payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs\" (UID: \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:35.936836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.936796 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht"] Nov 03 23:01:35.937668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:35.937641 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:36.007306 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.006873 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:36.265065 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.265031 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs"] Nov 03 23:01:36.265867 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:36.265778 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c3f9f2_b9ee_4605_9854_2e5778c4ad9c.slice/crio-a6eea19ef15cf8db9364548559678debc932b2e57ab977e67a991052eb9fd144 WatchSource:0}: Error finding container a6eea19ef15cf8db9364548559678debc932b2e57ab977e67a991052eb9fd144: Status 404 returned error can't find the container with id a6eea19ef15cf8db9364548559678debc932b2e57ab977e67a991052eb9fd144 Nov 03 23:01:36.320845 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.320808 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" event={"ID":"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c","Type":"ContainerStarted","Data":"a6eea19ef15cf8db9364548559678debc932b2e57ab977e67a991052eb9fd144"} Nov 03 23:01:36.359775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.359750 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7"] Nov 03 23:01:36.359952 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:36.359933 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad1242d_05c2_4478_84fb_d3fe522001a3.slice/crio-6345b2b3168588300d25b8d30317682b9c8d283784c71f9aad3b8cd3bf6a33b0 WatchSource:0}: Error finding container 6345b2b3168588300d25b8d30317682b9c8d283784c71f9aad3b8cd3bf6a33b0: Status 404 returned error can't find the container with id 6345b2b3168588300d25b8d30317682b9c8d283784c71f9aad3b8cd3bf6a33b0 Nov 03 23:01:36.751505 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.751472 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj"] Nov 03 23:01:36.757192 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.757165 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:36.771785 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.771691 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d8a3d1\"/\"openshift-service-ca.crt\"" Nov 03 23:01:36.771949 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.771834 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d8a3d1\"/\"kube-root-ca.crt\"" Nov 03 23:01:36.777470 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.777440 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj"] Nov 03 23:01:36.839609 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.839576 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d8a3d1\"/\"default-dockercfg-qbcsv\"" Nov 03 23:01:36.839798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.839742 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d8a3d1\"/\"rate-test-model-d8a3d1-kserve-self-signed-certs\"" Nov 03 23:01:36.920565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.920535 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-tmp-dir\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:36.920749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.920600 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-home\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:36.920749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.920736 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-model-cache\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:36.920841 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.920823 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-dshm\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:36.920877 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.920851 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d08ad03-a13d-4b82-a25d-00f821e29725-tls-certs\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:36.920922 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:36.920877 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpwhq\" (UniqueName: \"kubernetes.io/projected/7d08ad03-a13d-4b82-a25d-00f821e29725-kube-api-access-rpwhq\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.022474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.022362 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-tmp-dir\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.022669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.022470 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-home\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.022669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.022576 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-model-cache\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.022803 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.022656 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-dshm\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.022872 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.022798 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d08ad03-a13d-4b82-a25d-00f821e29725-tls-certs\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.022872 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.022826 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpwhq\" (UniqueName: \"kubernetes.io/projected/7d08ad03-a13d-4b82-a25d-00f821e29725-kube-api-access-rpwhq\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.023340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.022916 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-tmp-dir\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.023475 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.023363 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-home\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.023704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.023677 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-model-cache\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.026348 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.026326 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d08ad03-a13d-4b82-a25d-00f821e29725-tls-certs\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.027382 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.027351 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-dshm\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.068316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.063138 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpwhq\" (UniqueName: \"kubernetes.io/projected/7d08ad03-a13d-4b82-a25d-00f821e29725-kube-api-access-rpwhq\") pod \"rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.076633 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.076593 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:37.106318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.102560 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt"] Nov 03 23:01:37.114929 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.112667 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.129588 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.126589 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-162d7b-openshift-default-dockercfg-k7dhk\"" Nov 03 23:01:37.147778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.147746 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt"] Nov 03 23:01:37.227963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.227415 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istiod-ca-cert\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.227963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.227467 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-envoy\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.227963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.227488 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-podinfo\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.227963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.227561 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-token\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.227963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.227620 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-socket\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.227963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.227693 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-certs\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.227963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.227745 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-data\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.227963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.227774 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-credential-socket\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.227963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.227793 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5lt\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-kube-api-access-fz5lt\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.328996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.328871 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-data\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.328996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.328931 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-credential-socket\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.328996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.328961 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5lt\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-kube-api-access-fz5lt\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.329335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.329030 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istiod-ca-cert\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.329335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.329069 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-envoy\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.329335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.329095 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-podinfo\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.329335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.329188 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-token\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.329335 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.329267 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-socket\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.329621 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.329386 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-certs\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.330417 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.330388 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-certs\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.330594 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.330570 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-credential-socket\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.330777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.330389 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istiod-ca-cert\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.330777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.330596 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-socket\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.331044 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.330667 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-data\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.332584 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.332565 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-envoy\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.333037 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.333020 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-podinfo\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.336586 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.335610 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" event={"ID":"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c","Type":"ContainerStarted","Data":"39c07874915c2ae97372f0df20fd3089493ca770ad15164a0ac35eeab1614a9a"} Nov 03 23:01:37.336586 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.335721 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:37.338198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.338172 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" event={"ID":"aad1242d-05c2-4478-84fb-d3fe522001a3","Type":"ContainerStarted","Data":"437bb3ce731edc21692aff2fa08af27d1ce29d9c5cfbc65279290ad5e184c386"} Nov 03 23:01:37.338329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.338220 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" event={"ID":"aad1242d-05c2-4478-84fb-d3fe522001a3","Type":"ContainerStarted","Data":"6345b2b3168588300d25b8d30317682b9c8d283784c71f9aad3b8cd3bf6a33b0"} Nov 03 23:01:37.338329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.338312 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:37.352533 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.352504 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5lt\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-kube-api-access-fz5lt\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.352701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.352683 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-token\") pod \"e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.380022 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.379975 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" podStartSLOduration=2.379961622 podStartE2EDuration="2.379961622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:37.378214486 +0000 UTC m=+2923.782155462" watchObservedRunningTime="2026-11-03 23:01:37.379961622 +0000 UTC m=+2923.783902626" Nov 03 23:01:37.417399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.417350 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" podStartSLOduration=2.417337743 podStartE2EDuration="2.417337743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:37.415590833 +0000 UTC m=+2923.819531808" watchObservedRunningTime="2026-11-03 23:01:37.417337743 +0000 UTC m=+2923.821278718" Nov 03 23:01:37.447348 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.447316 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj"] Nov 03 23:01:37.447698 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:37.447669 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d08ad03_a13d_4b82_a25d_00f821e29725.slice/crio-f577352653e1877050dbabfb4e6f20bac371dc105d5b3bba9d873c2218286ae2 WatchSource:0}: Error finding container f577352653e1877050dbabfb4e6f20bac371dc105d5b3bba9d873c2218286ae2: Status 404 returned error can't find the container with id f577352653e1877050dbabfb4e6f20bac371dc105d5b3bba9d873c2218286ae2 Nov 03 23:01:37.457526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.457504 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:37.785956 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.785926 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt"] Nov 03 23:01:37.786450 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:37.785936 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea5c865_19c0_4343_b2d5_f34a94171fa8.slice/crio-cae59d6922e27621c8273e235667b08de69e8c5f95f46f651cdc73a28f38b1ac WatchSource:0}: Error finding container cae59d6922e27621c8273e235667b08de69e8c5f95f46f651cdc73a28f38b1ac: Status 404 returned error can't find the container with id cae59d6922e27621c8273e235667b08de69e8c5f95f46f651cdc73a28f38b1ac Nov 03 23:01:37.789020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.788987 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:01:37.789112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.789053 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:01:37.789112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:37.789083 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:01:38.348264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:38.348219 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" event={"ID":"7d08ad03-a13d-4b82-a25d-00f821e29725","Type":"ContainerStarted","Data":"96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd"} Nov 03 23:01:38.348264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:38.348268 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" event={"ID":"7d08ad03-a13d-4b82-a25d-00f821e29725","Type":"ContainerStarted","Data":"f577352653e1877050dbabfb4e6f20bac371dc105d5b3bba9d873c2218286ae2"} Nov 03 23:01:38.350047 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:38.350018 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" event={"ID":"2ea5c865-19c0-4343-b2d5-f34a94171fa8","Type":"ContainerStarted","Data":"e603f5b193260da999995423ea21b8587b309ab0cff2d08b3afe86e10ad26a75"} Nov 03 23:01:38.350170 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:38.350075 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" event={"ID":"2ea5c865-19c0-4343-b2d5-f34a94171fa8","Type":"ContainerStarted","Data":"cae59d6922e27621c8273e235667b08de69e8c5f95f46f651cdc73a28f38b1ac"} Nov 03 23:01:38.388454 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:38.388406 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" podStartSLOduration=2.388392752 podStartE2EDuration="2.388392752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:38.3857068 +0000 UTC m=+2924.789647775" watchObservedRunningTime="2026-11-03 23:01:38.388392752 +0000 UTC m=+2924.792333756" Nov 03 23:01:38.430085 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:38.430035 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podStartSLOduration=1.4300214709999999 podStartE2EDuration="1.430021471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:38.426966537 +0000 UTC m=+2924.830907513" watchObservedRunningTime="2026-11-03 23:01:38.430021471 +0000 UTC m=+2924.833962445" Nov 03 23:01:38.458711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:38.458683 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:38.464618 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:38.464592 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:39.077379 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:39.077339 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:39.358190 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:39.358098 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:39.359153 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:39.359136 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:01:40.905050 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:40.905007 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" podUID="8e2ce3a1-52cd-440b-b120-4d3953b41bdf" containerName="payload-processing" containerID="cri-o://3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b" gracePeriod=25 Nov 03 23:01:40.938153 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:40.938119 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" podUID="7423e5ac-0e71-463a-b099-f7af3f31a1d8" containerName="payload-pre-processing" containerID="cri-o://184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b" gracePeriod=25 Nov 03 23:01:41.229312 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.229270 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:01:41.283712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.283677 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:01:41.291980 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.291947 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-plugins-config-volume\") pod \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\" (UID: \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\") " Nov 03 23:01:41.292170 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.292039 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-kube-api-access-pq8h7\" (UniqueName: \"kubernetes.io/projected/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-kube-api-access-pq8h7\") pod \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\" (UID: \"8e2ce3a1-52cd-440b-b120-4d3953b41bdf\") " Nov 03 23:01:41.292583 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.292549 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-plugins-config-volume" pod "8e2ce3a1-52cd-440b-b120-4d3953b41bdf" (UID: "8e2ce3a1-52cd-440b-b120-4d3953b41bdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:41.292892 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.292868 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:41.294540 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.294516 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-kube-api-access-pq8h7" pod "8e2ce3a1-52cd-440b-b120-4d3953b41bdf" (UID: "8e2ce3a1-52cd-440b-b120-4d3953b41bdf"). InnerVolumeSpecName "kube-api-access-pq8h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:41.376244 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.376177 4903 generic.go:356] "Generic (PLEG): container finished" podID="7423e5ac-0e71-463a-b099-f7af3f31a1d8" containerID="184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b" exitCode=0 Nov 03 23:01:41.376244 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.376229 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" Nov 03 23:01:41.376541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.376265 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" event={"ID":"7423e5ac-0e71-463a-b099-f7af3f31a1d8","Type":"ContainerDied","Data":"184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b"} Nov 03 23:01:41.376541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.376342 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht" event={"ID":"7423e5ac-0e71-463a-b099-f7af3f31a1d8","Type":"ContainerDied","Data":"d9e8fbad43f848e9901aa2c6ddd8f1e70fd3197c14abaac1007b0f8db004d8c5"} Nov 03 23:01:41.376541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.376373 4903 scope.go:122] "RemoveContainer" containerID="184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b" Nov 03 23:01:41.377841 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.377818 4903 generic.go:356] "Generic (PLEG): container finished" podID="8e2ce3a1-52cd-440b-b120-4d3953b41bdf" containerID="3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b" exitCode=0 Nov 03 23:01:41.377977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.377866 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" Nov 03 23:01:41.377977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.377903 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" event={"ID":"8e2ce3a1-52cd-440b-b120-4d3953b41bdf","Type":"ContainerDied","Data":"3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b"} Nov 03 23:01:41.377977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.377951 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5" event={"ID":"8e2ce3a1-52cd-440b-b120-4d3953b41bdf","Type":"ContainerDied","Data":"9e62bb12a3e0ec19018257f99a2300013e151829e97e64a4292fee54ebe88cba"} Nov 03 23:01:41.387618 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.387597 4903 scope.go:122] "RemoveContainer" containerID="184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b" Nov 03 23:01:41.387893 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:41.387876 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b\": container with ID starting with 184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b not found: ID does not exist" containerID="184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b" Nov 03 23:01:41.387954 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.387899 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b"} err="failed to get container status \"184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b\": rpc error: code = NotFound desc = could not find container \"184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b\": container with ID starting with 184a06b3455a994bf50a57ab9e47d96474df8beff2c196233862b8f63861f17b not found: ID does not exist" Nov 03 23:01:41.387954 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.387916 4903 scope.go:122] "RemoveContainer" containerID="3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b" Nov 03 23:01:41.394027 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.394005 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7423e5ac-0e71-463a-b099-f7af3f31a1d8-kube-api-access-zx8f6\" (UniqueName: \"kubernetes.io/projected/7423e5ac-0e71-463a-b099-f7af3f31a1d8-kube-api-access-zx8f6\") pod \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\" (UID: \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\") " Nov 03 23:01:41.394143 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.394044 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7423e5ac-0e71-463a-b099-f7af3f31a1d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423e5ac-0e71-463a-b099-f7af3f31a1d8-plugins-config-volume\") pod \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\" (UID: \"7423e5ac-0e71-463a-b099-f7af3f31a1d8\") " Nov 03 23:01:41.394487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.394462 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7423e5ac-0e71-463a-b099-f7af3f31a1d8-plugins-config-volume" pod "7423e5ac-0e71-463a-b099-f7af3f31a1d8" (UID: "7423e5ac-0e71-463a-b099-f7af3f31a1d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:41.394487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.394479 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq8h7\" (UniqueName: \"kubernetes.io/projected/8e2ce3a1-52cd-440b-b120-4d3953b41bdf-kube-api-access-pq8h7\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:41.396236 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.396213 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7423e5ac-0e71-463a-b099-f7af3f31a1d8-kube-api-access-zx8f6" pod "7423e5ac-0e71-463a-b099-f7af3f31a1d8" (UID: "7423e5ac-0e71-463a-b099-f7af3f31a1d8"). InnerVolumeSpecName "kube-api-access-zx8f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:41.398520 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.398504 4903 scope.go:122] "RemoveContainer" containerID="3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b" Nov 03 23:01:41.398839 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:41.398817 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b\": container with ID starting with 3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b not found: ID does not exist" containerID="3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b" Nov 03 23:01:41.398902 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.398847 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b"} err="failed to get container status \"3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b\": rpc error: code = NotFound desc = could not find container \"3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b\": container with ID starting with 3622fcf842f1240fb5d888dc7f329d5bff113d63f726f4c64d563199978e517b not found: ID does not exist" Nov 03 23:01:41.430870 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.430804 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5"] Nov 03 23:01:41.440325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.440284 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4b994-65b8bd9f54-jszn5"] Nov 03 23:01:41.496105 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.496077 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8f6\" (UniqueName: \"kubernetes.io/projected/7423e5ac-0e71-463a-b099-f7af3f31a1d8-kube-api-access-zx8f6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:41.496105 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.496098 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423e5ac-0e71-463a-b099-f7af3f31a1d8-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:41.723630 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.723600 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht"] Nov 03 23:01:41.735594 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.735562 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2ce3a1-52cd-440b-b120-4d3953b41bdf" path="/var/lib/kubelet/pods/8e2ce3a1-52cd-440b-b120-4d3953b41bdf/volumes" Nov 03 23:01:41.735935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:41.735920 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-7d779cd5bc-sw2ht"] Nov 03 23:01:42.623251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.623212 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp"] Nov 03 23:01:42.624419 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.624395 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7423e5ac-0e71-463a-b099-f7af3f31a1d8" containerName="payload-pre-processing" Nov 03 23:01:42.624482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.624422 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7423e5ac-0e71-463a-b099-f7af3f31a1d8" containerName="payload-pre-processing" Nov 03 23:01:42.624482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.624446 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8e2ce3a1-52cd-440b-b120-4d3953b41bdf" containerName="payload-processing" Nov 03 23:01:42.624482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.624453 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2ce3a1-52cd-440b-b120-4d3953b41bdf" containerName="payload-processing" Nov 03 23:01:42.624667 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.624654 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7423e5ac-0e71-463a-b099-f7af3f31a1d8" containerName="payload-pre-processing" Nov 03 23:01:42.624704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.624672 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2ce3a1-52cd-440b-b120-4d3953b41bdf" containerName="payload-processing" Nov 03 23:01:42.630629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.630608 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:42.665690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.650104 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-162d7b-dockercfg-z5qj8\"" Nov 03 23:01:42.665690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.653833 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp"] Nov 03 23:01:42.665690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.656686 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-162d7b\"" Nov 03 23:01:42.713014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.712978 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfb70f9-51cc-4249-b088-af813d7b54dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp\" (UID: \"4dfb70f9-51cc-4249-b088-af813d7b54dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:42.713380 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.713350 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf8p6\" (UniqueName: \"kubernetes.io/projected/4dfb70f9-51cc-4249-b088-af813d7b54dc-kube-api-access-cf8p6\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp\" (UID: \"4dfb70f9-51cc-4249-b088-af813d7b54dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:42.760315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.747597 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh"] Nov 03 23:01:42.836347 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.816612 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:01:42.836347 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.825081 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfb70f9-51cc-4249-b088-af813d7b54dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp\" (UID: \"4dfb70f9-51cc-4249-b088-af813d7b54dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:42.836347 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.825184 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf8p6\" (UniqueName: \"kubernetes.io/projected/4dfb70f9-51cc-4249-b088-af813d7b54dc-kube-api-access-cf8p6\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp\" (UID: \"4dfb70f9-51cc-4249-b088-af813d7b54dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:42.836347 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.835081 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfb70f9-51cc-4249-b088-af813d7b54dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp\" (UID: \"4dfb70f9-51cc-4249-b088-af813d7b54dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:42.860146 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.860113 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf8p6\" (UniqueName: \"kubernetes.io/projected/4dfb70f9-51cc-4249-b088-af813d7b54dc-kube-api-access-cf8p6\") pod \"payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp\" (UID: \"4dfb70f9-51cc-4249-b088-af813d7b54dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:42.892829 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.892754 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh"] Nov 03 23:01:42.927354 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.927317 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkbh\" (UniqueName: \"kubernetes.io/projected/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-kube-api-access-5lkbh\") pod \"payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh\" (UID: \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:01:42.927713 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.927688 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh\" (UID: \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:01:42.950414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:42.950384 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:43.032967 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.031420 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkbh\" (UniqueName: \"kubernetes.io/projected/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-kube-api-access-5lkbh\") pod \"payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh\" (UID: \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:01:43.032967 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.031536 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh\" (UID: \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:01:43.032967 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.032553 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh\" (UID: \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:01:43.055681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.055650 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkbh\" (UniqueName: \"kubernetes.io/projected/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-kube-api-access-5lkbh\") pod \"payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh\" (UID: \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:01:43.195235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.195201 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:01:43.286183 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.284920 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp"] Nov 03 23:01:43.403353 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.403287 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" event={"ID":"4dfb70f9-51cc-4249-b088-af813d7b54dc","Type":"ContainerStarted","Data":"8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35"} Nov 03 23:01:43.403508 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.403373 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" event={"ID":"4dfb70f9-51cc-4249-b088-af813d7b54dc","Type":"ContainerStarted","Data":"071a8e70fe09e8b2b23bc3b5d7312026c7a6ceec6bd900899d30bb8108c83356"} Nov 03 23:01:43.403572 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.403549 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:43.459097 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.458985 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" podStartSLOduration=1.458971394 podStartE2EDuration="1.458971394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:43.452539839 +0000 UTC m=+2929.856480859" watchObservedRunningTime="2026-11-03 23:01:43.458971394 +0000 UTC m=+2929.862912368" Nov 03 23:01:43.521587 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.521555 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh"] Nov 03 23:01:43.522715 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:43.522676 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b50a9c0_0dc4_4b45_9ce4_efc3182d7450.slice/crio-1a1da6bda4d8af663be63ec6f36ceef6d5efebe24af6d497fa6e1a7e5f49021e WatchSource:0}: Error finding container 1a1da6bda4d8af663be63ec6f36ceef6d5efebe24af6d497fa6e1a7e5f49021e: Status 404 returned error can't find the container with id 1a1da6bda4d8af663be63ec6f36ceef6d5efebe24af6d497fa6e1a7e5f49021e Nov 03 23:01:43.737736 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:43.737636 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7423e5ac-0e71-463a-b099-f7af3f31a1d8" path="/var/lib/kubelet/pods/7423e5ac-0e71-463a-b099-f7af3f31a1d8/volumes" Nov 03 23:01:44.414482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:44.414444 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" event={"ID":"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450","Type":"ContainerStarted","Data":"2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0"} Nov 03 23:01:44.414704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:44.414521 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" event={"ID":"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450","Type":"ContainerStarted","Data":"1a1da6bda4d8af663be63ec6f36ceef6d5efebe24af6d497fa6e1a7e5f49021e"} Nov 03 23:01:44.414704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:44.414548 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:01:44.454413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:44.454365 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" podStartSLOduration=2.45435141 podStartE2EDuration="2.45435141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:44.450525064 +0000 UTC m=+2930.854466053" watchObservedRunningTime="2026-11-03 23:01:44.45435141 +0000 UTC m=+2930.858292381" Nov 03 23:01:45.234962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.234922 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m"] Nov 03 23:01:45.240657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.240635 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.256442 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.256416 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 23:01:45.256555 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.256464 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 23:01:45.256555 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.256479 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 23:01:45.256555 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.256513 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h854s\"" Nov 03 23:01:45.272053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.272020 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m"] Nov 03 23:01:45.366234 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.366202 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74fd6ab-74b5-4b05-b339-6c20da28e431-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.366438 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.366264 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthll\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-kube-api-access-fthll\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.366438 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.366388 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.366523 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.366500 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.366557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.366539 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.366632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.366613 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.366695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.366654 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.467683 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.467650 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.467860 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.467780 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74fd6ab-74b5-4b05-b339-6c20da28e431-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.467917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.467864 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthll\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-kube-api-access-fthll\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.467917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.467892 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.468037 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.467994 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.468103 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.468048 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.468159 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.468104 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.468814 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.468782 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.470251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.470220 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74fd6ab-74b5-4b05-b339-6c20da28e431-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.470486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.470464 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.470565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.470489 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.470565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.470519 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.489804 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.489744 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthll\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-kube-api-access-fthll\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.489935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.489912 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mzm2m\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.554847 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.554816 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:45.882496 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:45.882465 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74fd6ab_74b5_4b05_b339_6c20da28e431.slice/crio-8ead431be2ca59cd568584123ac708a4dee55c4e867def4cdc1598b024f4604b WatchSource:0}: Error finding container 8ead431be2ca59cd568584123ac708a4dee55c4e867def4cdc1598b024f4604b: Status 404 returned error can't find the container with id 8ead431be2ca59cd568584123ac708a4dee55c4e867def4cdc1598b024f4604b Nov 03 23:01:45.883157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.883130 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m"] Nov 03 23:01:45.885579 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.885471 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:01:45.885579 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:45.885538 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:01:46.434741 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:46.434641 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" event={"ID":"e74fd6ab-74b5-4b05-b339-6c20da28e431","Type":"ContainerStarted","Data":"4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502"} Nov 03 23:01:46.434741 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:46.434709 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" event={"ID":"e74fd6ab-74b5-4b05-b339-6c20da28e431","Type":"ContainerStarted","Data":"8ead431be2ca59cd568584123ac708a4dee55c4e867def4cdc1598b024f4604b"} Nov 03 23:01:46.434741 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:46.434736 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:46.436692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:46.436665 4903 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mzm2m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:01:46.436861 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:46.436719 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" podUID="e74fd6ab-74b5-4b05-b339-6c20da28e431" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:01:46.486089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:46.486025 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" podStartSLOduration=1.4860040460000001 podStartE2EDuration="1.486004046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:46.475449071 +0000 UTC m=+2932.879390040" watchObservedRunningTime="2026-11-03 23:01:46.486004046 +0000 UTC m=+2932.889945029" Nov 03 23:01:47.077640 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.077603 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:47.090673 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.090639 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:47.261795 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.261756 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg"] Nov 03 23:01:47.266756 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.266720 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.284184 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.283002 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:01:47.284412 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.284390 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r"] Nov 03 23:01:47.288923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.287373 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:01:47.288923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.287473 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:01:47.290248 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.290216 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.290986 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.290962 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg"] Nov 03 23:01:47.309772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.309729 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r"] Nov 03 23:01:47.397163 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.397125 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:01:47.397383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.397254 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82082c7d-s5q6r\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.397383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.397361 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82082c7d-s5q6r\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.400028 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.397640 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b2363a3-sksbg\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.400028 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.397689 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrlr4\" (UniqueName: \"kubernetes.io/projected/fca80af2-fc99-4d77-9c90-7427dda82697-kube-api-access-zrlr4\") pod \"maas-api-revoke-sub-82082c7d-s5q6r\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.400028 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.397738 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b2363a3-sksbg\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.400028 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.397770 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdp7v\" (UniqueName: \"kubernetes.io/projected/11d3d808-44b5-42da-8cf6-14e48559be90-kube-api-access-qdp7v\") pod \"maas-api-revoke-sub-0b2363a3-sksbg\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.445797 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.445767 4903 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mzm2m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:01:47.446275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.445823 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" podUID="e74fd6ab-74b5-4b05-b339-6c20da28e431" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:01:47.464712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.464679 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:01:47.499074 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.499042 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b2363a3-sksbg\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.499267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.499092 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrlr4\" (UniqueName: \"kubernetes.io/projected/fca80af2-fc99-4d77-9c90-7427dda82697-kube-api-access-zrlr4\") pod \"maas-api-revoke-sub-82082c7d-s5q6r\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.499267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.499121 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b2363a3-sksbg\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.499267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.499142 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdp7v\" (UniqueName: \"kubernetes.io/projected/11d3d808-44b5-42da-8cf6-14e48559be90-kube-api-access-qdp7v\") pod \"maas-api-revoke-sub-0b2363a3-sksbg\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.499490 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.499283 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82082c7d-s5q6r\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.499490 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.499366 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82082c7d-s5q6r\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.500144 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.500117 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82082c7d-s5q6r\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.500586 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.500564 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b2363a3-sksbg\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.501777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.501757 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b2363a3-sksbg\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.502063 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.502046 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82082c7d-s5q6r\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.527366 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.527338 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdp7v\" (UniqueName: \"kubernetes.io/projected/11d3d808-44b5-42da-8cf6-14e48559be90-kube-api-access-qdp7v\") pod \"maas-api-revoke-sub-0b2363a3-sksbg\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.529783 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.529758 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrlr4\" (UniqueName: \"kubernetes.io/projected/fca80af2-fc99-4d77-9c90-7427dda82697-kube-api-access-zrlr4\") pod \"maas-api-revoke-sub-82082c7d-s5q6r\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.586376 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.586339 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:47.647962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.647924 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:47.910861 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:47.910833 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg"] Nov 03 23:01:47.911136 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:47.911113 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d3d808_44b5_42da_8cf6_14e48559be90.slice/crio-d80aade689f2ce696d38c799e8e595069fb80de8855b30a6b8cc873e70e058bf WatchSource:0}: Error finding container d80aade689f2ce696d38c799e8e595069fb80de8855b30a6b8cc873e70e058bf: Status 404 returned error can't find the container with id d80aade689f2ce696d38c799e8e595069fb80de8855b30a6b8cc873e70e058bf Nov 03 23:01:48.039142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:48.039112 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r"] Nov 03 23:01:48.039623 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:01:48.039598 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca80af2_fc99_4d77_9c90_7427dda82697.slice/crio-93cbf64451dbf7c3b1e04a8d91dcc94f4f067faddffdee4e8bdaa6199665ec55 WatchSource:0}: Error finding container 93cbf64451dbf7c3b1e04a8d91dcc94f4f067faddffdee4e8bdaa6199665ec55: Status 404 returned error can't find the container with id 93cbf64451dbf7c3b1e04a8d91dcc94f4f067faddffdee4e8bdaa6199665ec55 Nov 03 23:01:48.353374 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:48.353349 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:01:48.354337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:48.354316 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:01:48.385721 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:48.385687 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:01:48.462599 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:48.460687 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" event={"ID":"11d3d808-44b5-42da-8cf6-14e48559be90","Type":"ContainerStarted","Data":"4a465f803943fd48401addecc14fbfadaaeedfa2e1de81f54c15a4cf9074e2dc"} Nov 03 23:01:48.462599 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:48.460779 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" event={"ID":"11d3d808-44b5-42da-8cf6-14e48559be90","Type":"ContainerStarted","Data":"d80aade689f2ce696d38c799e8e595069fb80de8855b30a6b8cc873e70e058bf"} Nov 03 23:01:48.464227 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:48.464201 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" event={"ID":"fca80af2-fc99-4d77-9c90-7427dda82697","Type":"ContainerStarted","Data":"a288db4429a55837846d2925455f32efe028670c0f739f3846ef05ecdfab213c"} Nov 03 23:01:48.464392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:48.464243 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" event={"ID":"fca80af2-fc99-4d77-9c90-7427dda82697","Type":"ContainerStarted","Data":"93cbf64451dbf7c3b1e04a8d91dcc94f4f067faddffdee4e8bdaa6199665ec55"} Nov 03 23:01:48.503575 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:48.503523 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" podStartSLOduration=1.503510083 podStartE2EDuration="1.503510083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:48.498970094 +0000 UTC m=+2934.902911070" watchObservedRunningTime="2026-11-03 23:01:48.503510083 +0000 UTC m=+2934.907451059" Nov 03 23:01:48.539094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:48.538990 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" podStartSLOduration=1.538974174 podStartE2EDuration="1.538974174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:01:48.534709578 +0000 UTC m=+2934.938650549" watchObservedRunningTime="2026-11-03 23:01:48.538974174 +0000 UTC m=+2934.942915150" Nov 03 23:01:49.475733 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:49.475700 4903 generic.go:356] "Generic (PLEG): container finished" podID="11d3d808-44b5-42da-8cf6-14e48559be90" containerID="4a465f803943fd48401addecc14fbfadaaeedfa2e1de81f54c15a4cf9074e2dc" exitCode=0 Nov 03 23:01:49.476172 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:49.475795 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" event={"ID":"11d3d808-44b5-42da-8cf6-14e48559be90","Type":"ContainerDied","Data":"4a465f803943fd48401addecc14fbfadaaeedfa2e1de81f54c15a4cf9074e2dc"} Nov 03 23:01:49.477567 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:49.477545 4903 generic.go:356] "Generic (PLEG): container finished" podID="fca80af2-fc99-4d77-9c90-7427dda82697" containerID="a288db4429a55837846d2925455f32efe028670c0f739f3846ef05ecdfab213c" exitCode=0 Nov 03 23:01:49.477678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:49.477590 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" event={"ID":"fca80af2-fc99-4d77-9c90-7427dda82697","Type":"ContainerDied","Data":"a288db4429a55837846d2925455f32efe028670c0f739f3846ef05ecdfab213c"} Nov 03 23:01:50.446494 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.446465 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:01:50.718925 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.718897 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:50.723241 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.723218 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:50.859665 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.859635 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-openshift-ca\") pod \"11d3d808-44b5-42da-8cf6-14e48559be90\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " Nov 03 23:01:50.859847 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.859698 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca80af2-fc99-4d77-9c90-7427dda82697-kube-api-access-zrlr4\" (UniqueName: \"kubernetes.io/projected/fca80af2-fc99-4d77-9c90-7427dda82697-kube-api-access-zrlr4\") pod \"fca80af2-fc99-4d77-9c90-7427dda82697\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " Nov 03 23:01:50.859847 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.859819 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-openshift-ca\") pod \"fca80af2-fc99-4d77-9c90-7427dda82697\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " Nov 03 23:01:50.859958 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.859938 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-xks-ca\") pod \"fca80af2-fc99-4d77-9c90-7427dda82697\" (UID: \"fca80af2-fc99-4d77-9c90-7427dda82697\") " Nov 03 23:01:50.859999 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.859980 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-xks-ca\") pod \"11d3d808-44b5-42da-8cf6-14e48559be90\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " Nov 03 23:01:50.860051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.860021 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d3d808-44b5-42da-8cf6-14e48559be90-kube-api-access-qdp7v\" (UniqueName: \"kubernetes.io/projected/11d3d808-44b5-42da-8cf6-14e48559be90-kube-api-access-qdp7v\") pod \"11d3d808-44b5-42da-8cf6-14e48559be90\" (UID: \"11d3d808-44b5-42da-8cf6-14e48559be90\") " Nov 03 23:01:50.860131 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.860091 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-openshift-ca" pod "11d3d808-44b5-42da-8cf6-14e48559be90" (UID: "11d3d808-44b5-42da-8cf6-14e48559be90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:50.860211 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.860162 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-openshift-ca" pod "fca80af2-fc99-4d77-9c90-7427dda82697" (UID: "fca80af2-fc99-4d77-9c90-7427dda82697"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:50.860826 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.860800 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:50.860973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.860831 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:50.862134 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.862109 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-xks-ca" pod "fca80af2-fc99-4d77-9c90-7427dda82697" (UID: "fca80af2-fc99-4d77-9c90-7427dda82697"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:50.862262 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.862248 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d3d808-44b5-42da-8cf6-14e48559be90-kube-api-access-qdp7v" pod "11d3d808-44b5-42da-8cf6-14e48559be90" (UID: "11d3d808-44b5-42da-8cf6-14e48559be90"). InnerVolumeSpecName "kube-api-access-qdp7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:50.862466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.862443 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-xks-ca" pod "11d3d808-44b5-42da-8cf6-14e48559be90" (UID: "11d3d808-44b5-42da-8cf6-14e48559be90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:01:50.862560 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.862536 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca80af2-fc99-4d77-9c90-7427dda82697-kube-api-access-zrlr4" pod "fca80af2-fc99-4d77-9c90-7427dda82697" (UID: "fca80af2-fc99-4d77-9c90-7427dda82697"). InnerVolumeSpecName "kube-api-access-zrlr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:50.962948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.962860 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca80af2-fc99-4d77-9c90-7427dda82697-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:50.962948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.962888 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11d3d808-44b5-42da-8cf6-14e48559be90-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:50.962948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.962897 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdp7v\" (UniqueName: \"kubernetes.io/projected/11d3d808-44b5-42da-8cf6-14e48559be90-kube-api-access-qdp7v\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:50.962948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:50.962908 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrlr4\" (UniqueName: \"kubernetes.io/projected/fca80af2-fc99-4d77-9c90-7427dda82697-kube-api-access-zrlr4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:51.496436 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:51.496405 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" Nov 03 23:01:51.496632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:51.496432 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg" event={"ID":"11d3d808-44b5-42da-8cf6-14e48559be90","Type":"ContainerDied","Data":"d80aade689f2ce696d38c799e8e595069fb80de8855b30a6b8cc873e70e058bf"} Nov 03 23:01:51.496632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:51.496484 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d80aade689f2ce696d38c799e8e595069fb80de8855b30a6b8cc873e70e058bf" Nov 03 23:01:51.497925 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:51.497906 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" Nov 03 23:01:51.498033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:51.497941 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r" event={"ID":"fca80af2-fc99-4d77-9c90-7427dda82697","Type":"ContainerDied","Data":"93cbf64451dbf7c3b1e04a8d91dcc94f4f067faddffdee4e8bdaa6199665ec55"} Nov 03 23:01:51.498033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:51.497984 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93cbf64451dbf7c3b1e04a8d91dcc94f4f067faddffdee4e8bdaa6199665ec55" Nov 03 23:01:52.716118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:52.716078 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m"] Nov 03 23:01:52.716585 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:52.716339 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" podUID="8a3373dd-b15e-490d-9a62-8b271356e69f" containerName="payload-processing" containerID="cri-o://335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096" gracePeriod=30 Nov 03 23:01:52.737166 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:52.737136 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx"] Nov 03 23:01:52.737409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:52.737381 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" podUID="83ed7ec5-6b47-4934-9ef6-135fda1e2557" containerName="payload-pre-processing" containerID="cri-o://9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f" gracePeriod=30 Nov 03 23:01:53.050855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.050826 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:01:53.087021 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.086993 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:01:53.199316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.197778 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ed7ec5-6b47-4934-9ef6-135fda1e2557-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ed7ec5-6b47-4934-9ef6-135fda1e2557-plugins-config-volume\") pod \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\" (UID: \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\") " Nov 03 23:01:53.199316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.198314 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ed7ec5-6b47-4934-9ef6-135fda1e2557-plugins-config-volume" pod "83ed7ec5-6b47-4934-9ef6-135fda1e2557" (UID: "83ed7ec5-6b47-4934-9ef6-135fda1e2557"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:53.199316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.198445 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ed7ec5-6b47-4934-9ef6-135fda1e2557-kube-api-access-h8r79\" (UniqueName: \"kubernetes.io/projected/83ed7ec5-6b47-4934-9ef6-135fda1e2557-kube-api-access-h8r79\") pod \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\" (UID: \"83ed7ec5-6b47-4934-9ef6-135fda1e2557\") " Nov 03 23:01:53.199316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.198715 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3373dd-b15e-490d-9a62-8b271356e69f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3373dd-b15e-490d-9a62-8b271356e69f-plugins-config-volume\") pod \"8a3373dd-b15e-490d-9a62-8b271356e69f\" (UID: \"8a3373dd-b15e-490d-9a62-8b271356e69f\") " Nov 03 23:01:53.199316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.198748 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3373dd-b15e-490d-9a62-8b271356e69f-kube-api-access-f77mk\" (UniqueName: \"kubernetes.io/projected/8a3373dd-b15e-490d-9a62-8b271356e69f-kube-api-access-f77mk\") pod \"8a3373dd-b15e-490d-9a62-8b271356e69f\" (UID: \"8a3373dd-b15e-490d-9a62-8b271356e69f\") " Nov 03 23:01:53.199316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.199271 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ed7ec5-6b47-4934-9ef6-135fda1e2557-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:53.200888 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.200072 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3373dd-b15e-490d-9a62-8b271356e69f-plugins-config-volume" pod "8a3373dd-b15e-490d-9a62-8b271356e69f" (UID: "8a3373dd-b15e-490d-9a62-8b271356e69f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:01:53.204332 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.203399 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3373dd-b15e-490d-9a62-8b271356e69f-kube-api-access-f77mk" pod "8a3373dd-b15e-490d-9a62-8b271356e69f" (UID: "8a3373dd-b15e-490d-9a62-8b271356e69f"). InnerVolumeSpecName "kube-api-access-f77mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:53.213426 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.213401 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ed7ec5-6b47-4934-9ef6-135fda1e2557-kube-api-access-h8r79" pod "83ed7ec5-6b47-4934-9ef6-135fda1e2557" (UID: "83ed7ec5-6b47-4934-9ef6-135fda1e2557"). InnerVolumeSpecName "kube-api-access-h8r79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:01:53.300613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.300532 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3373dd-b15e-490d-9a62-8b271356e69f-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:53.300613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.300572 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f77mk\" (UniqueName: \"kubernetes.io/projected/8a3373dd-b15e-490d-9a62-8b271356e69f-kube-api-access-f77mk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:53.300613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.300587 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8r79\" (UniqueName: \"kubernetes.io/projected/83ed7ec5-6b47-4934-9ef6-135fda1e2557-kube-api-access-h8r79\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:01:53.352667 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.352632 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx"] Nov 03 23:01:53.357016 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.356991 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e7a7-vrchx"] Nov 03 23:01:53.516868 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.516835 4903 generic.go:356] "Generic (PLEG): container finished" podID="8a3373dd-b15e-490d-9a62-8b271356e69f" containerID="335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096" exitCode=0 Nov 03 23:01:53.517071 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.516882 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" Nov 03 23:01:53.517071 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.516919 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" event={"ID":"8a3373dd-b15e-490d-9a62-8b271356e69f","Type":"ContainerDied","Data":"335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096"} Nov 03 23:01:53.517071 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.516970 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m" event={"ID":"8a3373dd-b15e-490d-9a62-8b271356e69f","Type":"ContainerDied","Data":"e08bc4d9820cf52009fe5b0305f7b9d377b6fb30e9d18ec0941fc95a3bbceacc"} Nov 03 23:01:53.517071 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.516999 4903 scope.go:122] "RemoveContainer" containerID="335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096" Nov 03 23:01:53.518558 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.518529 4903 generic.go:356] "Generic (PLEG): container finished" podID="83ed7ec5-6b47-4934-9ef6-135fda1e2557" containerID="9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f" exitCode=0 Nov 03 23:01:53.518690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.518583 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" Nov 03 23:01:53.518690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.518605 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" event={"ID":"83ed7ec5-6b47-4934-9ef6-135fda1e2557","Type":"ContainerDied","Data":"9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f"} Nov 03 23:01:53.518690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.518652 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx" event={"ID":"83ed7ec5-6b47-4934-9ef6-135fda1e2557","Type":"ContainerDied","Data":"23626a07369d69ac72dfa07f3d8fed1d506abf53c84eb888b0b84ec1ac350973"} Nov 03 23:01:53.533768 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.531023 4903 scope.go:122] "RemoveContainer" containerID="335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096" Nov 03 23:01:53.533768 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:53.531360 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096\": container with ID starting with 335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096 not found: ID does not exist" containerID="335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096" Nov 03 23:01:53.533768 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.531392 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096"} err="failed to get container status \"335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096\": rpc error: code = NotFound desc = could not find container \"335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096\": container with ID starting with 335e6aab0dfa76def5745fd4a711b5db3e418855d1cbf31e924b46e9897d6096 not found: ID does not exist" Nov 03 23:01:53.533768 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.531411 4903 scope.go:122] "RemoveContainer" containerID="9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f" Nov 03 23:01:53.570948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.570864 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m"] Nov 03 23:01:53.572147 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.572126 4903 scope.go:122] "RemoveContainer" containerID="9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f" Nov 03 23:01:53.573589 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:01:53.573559 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f\": container with ID starting with 9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f not found: ID does not exist" containerID="9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f" Nov 03 23:01:53.573701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.573599 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f"} err="failed to get container status \"9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f\": rpc error: code = NotFound desc = could not find container \"9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f\": container with ID starting with 9971cee622bb8beac3e6d3ec9f9aeb105fc9ed768b0f6ccd13de95aa9e0c821f not found: ID does not exist" Nov 03 23:01:53.576993 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.576968 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e7a7-5f4f86db5d-mnf4m"] Nov 03 23:01:53.609609 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.609568 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx"] Nov 03 23:01:53.616408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.616378 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e7a7-dd597bff5-j8vmx"] Nov 03 23:01:53.739594 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.739562 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5d0ae5-dca1-4104-8379-92e90cd60201" path="/var/lib/kubelet/pods/0d5d0ae5-dca1-4104-8379-92e90cd60201/volumes" Nov 03 23:01:53.740404 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.740379 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ed7ec5-6b47-4934-9ef6-135fda1e2557" path="/var/lib/kubelet/pods/83ed7ec5-6b47-4934-9ef6-135fda1e2557/volumes" Nov 03 23:01:53.740874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:53.740853 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3373dd-b15e-490d-9a62-8b271356e69f" path="/var/lib/kubelet/pods/8a3373dd-b15e-490d-9a62-8b271356e69f/volumes" Nov 03 23:01:54.417748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:54.417715 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:01:55.427736 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:01:55.427703 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:02:01.172193 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.172157 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld"] Nov 03 23:02:01.172800 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.172456 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" containerID="cri-o://4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876" gracePeriod=30 Nov 03 23:02:01.745321 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.745273 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w"] Nov 03 23:02:01.746275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746256 4903 cpu_manager.go:414] "containerMap: removing container" podUID="fca80af2-fc99-4d77-9c90-7427dda82697" containerName="revoke-keys" Nov 03 23:02:01.746371 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746277 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca80af2-fc99-4d77-9c90-7427dda82697" containerName="revoke-keys" Nov 03 23:02:01.746371 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746288 4903 cpu_manager.go:414] "containerMap: removing container" podUID="11d3d808-44b5-42da-8cf6-14e48559be90" containerName="revoke-keys" Nov 03 23:02:01.746371 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746309 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d3d808-44b5-42da-8cf6-14e48559be90" containerName="revoke-keys" Nov 03 23:02:01.746371 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746352 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8a3373dd-b15e-490d-9a62-8b271356e69f" containerName="payload-processing" Nov 03 23:02:01.746371 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746357 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3373dd-b15e-490d-9a62-8b271356e69f" containerName="payload-processing" Nov 03 23:02:01.746632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746396 4903 cpu_manager.go:414] "containerMap: removing container" podUID="83ed7ec5-6b47-4934-9ef6-135fda1e2557" containerName="payload-pre-processing" Nov 03 23:02:01.746632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746401 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ed7ec5-6b47-4934-9ef6-135fda1e2557" containerName="payload-pre-processing" Nov 03 23:02:01.746632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746518 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3373dd-b15e-490d-9a62-8b271356e69f" containerName="payload-processing" Nov 03 23:02:01.746632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746534 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ed7ec5-6b47-4934-9ef6-135fda1e2557" containerName="payload-pre-processing" Nov 03 23:02:01.746632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746542 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca80af2-fc99-4d77-9c90-7427dda82697" containerName="revoke-keys" Nov 03 23:02:01.746632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.746548 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d3d808-44b5-42da-8cf6-14e48559be90" containerName="revoke-keys" Nov 03 23:02:01.752269 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.752249 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:01.767029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.766999 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w"] Nov 03 23:02:01.771367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.771345 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:02:01.771367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.771360 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:02:01.789367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.789337 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:02:01.789520 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.789392 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:01.830936 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.830905 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:01.831094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.830961 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ls9\" (UniqueName: \"kubernetes.io/projected/cba195c6-6516-463d-a553-45081c6b7c71-kube-api-access-f8ls9\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:01.831278 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.831244 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:01.932804 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.932762 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:01.933005 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.932820 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ls9\" (UniqueName: \"kubernetes.io/projected/cba195c6-6516-463d-a553-45081c6b7c71-kube-api-access-f8ls9\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:01.933072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.933018 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:01.933746 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.933718 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:01.935395 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.935366 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:01.991135 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:01.991097 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ls9\" (UniqueName: \"kubernetes.io/projected/cba195c6-6516-463d-a553-45081c6b7c71-kube-api-access-f8ls9\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:02.069893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:02.069807 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:02.381680 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:02.381645 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w"] Nov 03 23:02:02.382107 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:02:02.381737 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba195c6_6516_463d_a553_45081c6b7c71.slice/crio-90fab8a6d7fbe08bca0696d41b32982391b9a125ef6c8a5029dc9badb916855d WatchSource:0}: Error finding container 90fab8a6d7fbe08bca0696d41b32982391b9a125ef6c8a5029dc9badb916855d: Status 404 returned error can't find the container with id 90fab8a6d7fbe08bca0696d41b32982391b9a125ef6c8a5029dc9badb916855d Nov 03 23:02:02.603621 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:02.603583 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" event={"ID":"cba195c6-6516-463d-a553-45081c6b7c71","Type":"ContainerStarted","Data":"90fab8a6d7fbe08bca0696d41b32982391b9a125ef6c8a5029dc9badb916855d"} Nov 03 23:02:02.965236 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:02.965148 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:02:02.965236 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:02.965217 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:02:03.612665 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:03.612577 4903 generic.go:356] "Generic (PLEG): container finished" podID="cba195c6-6516-463d-a553-45081c6b7c71" containerID="57d8df15e73b241f5e3bb95b9f05adb4dfb871e17271cd4045c2bfc3229a4dbb" exitCode=0 Nov 03 23:02:03.613120 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:03.612664 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" event={"ID":"cba195c6-6516-463d-a553-45081c6b7c71","Type":"ContainerDied","Data":"57d8df15e73b241f5e3bb95b9f05adb4dfb871e17271cd4045c2bfc3229a4dbb"} Nov 03 23:02:04.797150 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.797125 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:04.878319 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.878215 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-openshift-ca\") pod \"cba195c6-6516-463d-a553-45081c6b7c71\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " Nov 03 23:02:04.878319 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.878289 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba195c6-6516-463d-a553-45081c6b7c71-kube-api-access-f8ls9\" (UniqueName: \"kubernetes.io/projected/cba195c6-6516-463d-a553-45081c6b7c71-kube-api-access-f8ls9\") pod \"cba195c6-6516-463d-a553-45081c6b7c71\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " Nov 03 23:02:04.878516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.878357 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-xks-ca\") pod \"cba195c6-6516-463d-a553-45081c6b7c71\" (UID: \"cba195c6-6516-463d-a553-45081c6b7c71\") " Nov 03 23:02:04.878651 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.878625 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-openshift-ca" pod "cba195c6-6516-463d-a553-45081c6b7c71" (UID: "cba195c6-6516-463d-a553-45081c6b7c71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:04.879020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.878993 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:04.880656 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.880633 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba195c6-6516-463d-a553-45081c6b7c71-kube-api-access-f8ls9" pod "cba195c6-6516-463d-a553-45081c6b7c71" (UID: "cba195c6-6516-463d-a553-45081c6b7c71"). InnerVolumeSpecName "kube-api-access-f8ls9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:04.880769 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.880663 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-xks-ca" pod "cba195c6-6516-463d-a553-45081c6b7c71" (UID: "cba195c6-6516-463d-a553-45081c6b7c71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:04.980165 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.980132 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ls9\" (UniqueName: \"kubernetes.io/projected/cba195c6-6516-463d-a553-45081c6b7c71-kube-api-access-f8ls9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:04.980165 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:04.980161 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cba195c6-6516-463d-a553-45081c6b7c71-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:05.629869 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:05.629838 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" Nov 03 23:02:05.630054 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:05.629871 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w" event={"ID":"cba195c6-6516-463d-a553-45081c6b7c71","Type":"ContainerDied","Data":"90fab8a6d7fbe08bca0696d41b32982391b9a125ef6c8a5029dc9badb916855d"} Nov 03 23:02:05.630054 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:05.629918 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90fab8a6d7fbe08bca0696d41b32982391b9a125ef6c8a5029dc9badb916855d" Nov 03 23:02:06.455125 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.455069 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 23:02:06.602000 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.601964 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-kube-api-access-bfvnv\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-kube-api-access-bfvnv\") pod \"83503d8f-0a17-4c64-84af-d93426654e30\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " Nov 03 23:02:06.602276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602014 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-credential-socket\") pod \"83503d8f-0a17-4c64-84af-d93426654e30\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " Nov 03 23:02:06.602276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602035 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-envoy\") pod \"83503d8f-0a17-4c64-84af-d93426654e30\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " Nov 03 23:02:06.602276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602102 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83503d8f-0a17-4c64-84af-d93426654e30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83503d8f-0a17-4c64-84af-d93426654e30-istiod-ca-cert\") pod \"83503d8f-0a17-4c64-84af-d93426654e30\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " Nov 03 23:02:06.602276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602239 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-certs\") pod \"83503d8f-0a17-4c64-84af-d93426654e30\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " Nov 03 23:02:06.602625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602289 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-data\") pod \"83503d8f-0a17-4c64-84af-d93426654e30\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " Nov 03 23:02:06.602625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602355 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-credential-socket" pod "83503d8f-0a17-4c64-84af-d93426654e30" (UID: "83503d8f-0a17-4c64-84af-d93426654e30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:06.602625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602397 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-istio-token\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-istio-token\") pod \"83503d8f-0a17-4c64-84af-d93426654e30\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " Nov 03 23:02:06.602625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602438 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-socket\") pod \"83503d8f-0a17-4c64-84af-d93426654e30\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " Nov 03 23:02:06.602625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602516 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-certs" pod "83503d8f-0a17-4c64-84af-d93426654e30" (UID: "83503d8f-0a17-4c64-84af-d93426654e30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:06.602625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602524 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83503d8f-0a17-4c64-84af-d93426654e30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83503d8f-0a17-4c64-84af-d93426654e30-istio-podinfo\") pod \"83503d8f-0a17-4c64-84af-d93426654e30\" (UID: \"83503d8f-0a17-4c64-84af-d93426654e30\") " Nov 03 23:02:06.602625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602565 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83503d8f-0a17-4c64-84af-d93426654e30-istiod-ca-cert" pod "83503d8f-0a17-4c64-84af-d93426654e30" (UID: "83503d8f-0a17-4c64-84af-d93426654e30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:06.602625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602599 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-data" pod "83503d8f-0a17-4c64-84af-d93426654e30" (UID: "83503d8f-0a17-4c64-84af-d93426654e30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:06.603088 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.602831 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-socket" pod "83503d8f-0a17-4c64-84af-d93426654e30" (UID: "83503d8f-0a17-4c64-84af-d93426654e30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:06.603311 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.603270 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:06.603501 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.603474 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83503d8f-0a17-4c64-84af-d93426654e30-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:06.603501 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.603501 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:06.603689 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.603516 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:06.603689 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.603530 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:06.604401 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.604354 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-envoy" pod "83503d8f-0a17-4c64-84af-d93426654e30" (UID: "83503d8f-0a17-4c64-84af-d93426654e30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:02:06.604658 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.604635 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-kube-api-access-bfvnv" pod "83503d8f-0a17-4c64-84af-d93426654e30" (UID: "83503d8f-0a17-4c64-84af-d93426654e30"). InnerVolumeSpecName "kube-api-access-bfvnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:06.604767 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.604666 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-istio-token" pod "83503d8f-0a17-4c64-84af-d93426654e30" (UID: "83503d8f-0a17-4c64-84af-d93426654e30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:06.604826 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.604791 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83503d8f-0a17-4c64-84af-d93426654e30-istio-podinfo" pod "83503d8f-0a17-4c64-84af-d93426654e30" (UID: "83503d8f-0a17-4c64-84af-d93426654e30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:02:06.639792 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.639761 4903 generic.go:356] "Generic (PLEG): container finished" podID="83503d8f-0a17-4c64-84af-d93426654e30" containerID="4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876" exitCode=0 Nov 03 23:02:06.639961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.639824 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" Nov 03 23:02:06.639961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.639855 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" event={"ID":"83503d8f-0a17-4c64-84af-d93426654e30","Type":"ContainerDied","Data":"4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876"} Nov 03 23:02:06.639961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.639952 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld" event={"ID":"83503d8f-0a17-4c64-84af-d93426654e30","Type":"ContainerDied","Data":"7415ae54776c3091cfab100df09f33b69788c0cf35ade3146c7302b06da03b46"} Nov 03 23:02:06.640146 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.639984 4903 scope.go:122] "RemoveContainer" containerID="4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876" Nov 03 23:02:06.651602 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.651585 4903 scope.go:122] "RemoveContainer" containerID="b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30" Nov 03 23:02:06.661730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.661703 4903 scope.go:122] "RemoveContainer" containerID="4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876" Nov 03 23:02:06.661963 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:02:06.661945 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876\": container with ID starting with 4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876 not found: ID does not exist" containerID="4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876" Nov 03 23:02:06.662026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.661969 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876"} err="failed to get container status \"4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876\": rpc error: code = NotFound desc = could not find container \"4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876\": container with ID starting with 4f0b93fa3642d61e5981c19a466892b6d6741c565456ad5ddcac728d76e51876 not found: ID does not exist" Nov 03 23:02:06.662026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.661985 4903 scope.go:122] "RemoveContainer" containerID="b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30" Nov 03 23:02:06.662209 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:02:06.662189 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30\": container with ID starting with b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30 not found: ID does not exist" containerID="b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30" Nov 03 23:02:06.662261 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.662216 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30"} err="failed to get container status \"b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30\": rpc error: code = NotFound desc = could not find container \"b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30\": container with ID starting with b82b0d7e5b2419146ad71a516d06346d404d81cd6b9465a5f71cb5c626d54e30 not found: ID does not exist" Nov 03 23:02:06.691582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.691555 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld"] Nov 03 23:02:06.695576 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.695554 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-15e7a7-openshift-default-7446d84559-dxsld"] Nov 03 23:02:06.704701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.704679 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83503d8f-0a17-4c64-84af-d93426654e30-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:06.704701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.704701 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:06.704842 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.704709 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83503d8f-0a17-4c64-84af-d93426654e30-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:06.704842 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:06.704718 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfvnv\" (UniqueName: \"kubernetes.io/projected/83503d8f-0a17-4c64-84af-d93426654e30-kube-api-access-bfvnv\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:07.736621 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:07.736587 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83503d8f-0a17-4c64-84af-d93426654e30" path="/var/lib/kubelet/pods/83503d8f-0a17-4c64-84af-d93426654e30/volumes" Nov 03 23:02:16.118604 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.118569 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k"] Nov 03 23:02:16.119713 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.119684 4903 cpu_manager.go:414] "containerMap: removing container" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" Nov 03 23:02:16.119713 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.119712 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" Nov 03 23:02:16.119962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.119768 4903 cpu_manager.go:414] "containerMap: removing container" podUID="cba195c6-6516-463d-a553-45081c6b7c71" containerName="revoke-keys" Nov 03 23:02:16.119962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.119777 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba195c6-6516-463d-a553-45081c6b7c71" containerName="revoke-keys" Nov 03 23:02:16.119962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.119956 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" Nov 03 23:02:16.120135 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.119976 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba195c6-6516-463d-a553-45081c6b7c71" containerName="revoke-keys" Nov 03 23:02:16.127405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.127377 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:16.146986 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.146945 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k"] Nov 03 23:02:16.150044 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.149996 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a4b994\"" Nov 03 23:02:16.150044 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.150035 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a4b994-dockercfg-7q8pr\"" Nov 03 23:02:16.155582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.155150 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k\" (UID: \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:16.155582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.155365 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265xn\" (UniqueName: \"kubernetes.io/projected/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-kube-api-access-265xn\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k\" (UID: \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:16.256264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.256227 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k\" (UID: \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:16.256473 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.256324 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265xn\" (UniqueName: \"kubernetes.io/projected/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-kube-api-access-265xn\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k\" (UID: \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:16.257046 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.257024 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k\" (UID: \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:16.278117 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.278092 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-265xn\" (UniqueName: \"kubernetes.io/projected/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-kube-api-access-265xn\") pod \"payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k\" (UID: \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:16.454059 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.454022 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:16.760473 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:16.760439 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k"] Nov 03 23:02:16.760614 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:02:16.760481 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb02b4d_d081_4caf_82b6_c4f41ae31eb8.slice/crio-583a916aa7835149f1ebaeffbd54e6681fd6d7db1b8ee87206044eb179bf43d9 WatchSource:0}: Error finding container 583a916aa7835149f1ebaeffbd54e6681fd6d7db1b8ee87206044eb179bf43d9: Status 404 returned error can't find the container with id 583a916aa7835149f1ebaeffbd54e6681fd6d7db1b8ee87206044eb179bf43d9 Nov 03 23:02:17.745671 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:17.745633 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" event={"ID":"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8","Type":"ContainerStarted","Data":"4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5"} Nov 03 23:02:17.746138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:17.745697 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:17.746138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:17.745721 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" event={"ID":"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8","Type":"ContainerStarted","Data":"583a916aa7835149f1ebaeffbd54e6681fd6d7db1b8ee87206044eb179bf43d9"} Nov 03 23:02:17.782227 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:17.782168 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" podStartSLOduration=1.7821499790000002 podStartE2EDuration="1.782149979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:17.781995759 +0000 UTC m=+2964.185936735" watchObservedRunningTime="2026-11-03 23:02:17.782149979 +0000 UTC m=+2964.186090955" Nov 03 23:02:18.385454 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:02:18.385414 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:02:21.947336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:21.947306 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2_53bd4dbd-7d51-4454-a43b-b8f5680c3857/payload-processing/0.log" Nov 03 23:02:22.132606 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:22.132565 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:02:22.336588 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:22.336500 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:02:22.521678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:22.521651 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f_b2341ba1-d0ba-460a-ab1e-58a6fd17e217/payload-processing/0.log" Nov 03 23:02:22.698823 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:22.698799 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f_b2341ba1-d0ba-460a-ab1e-58a6fd17e217/payload-processing/0.log" Nov 03 23:02:23.428118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.428086 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4"] Nov 03 23:02:23.428826 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.428811 4903 cpu_manager.go:414] "containerMap: removing container" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" Nov 03 23:02:23.428872 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.428826 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" Nov 03 23:02:23.428947 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.428938 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="83503d8f-0a17-4c64-84af-d93426654e30" containerName="istio-proxy" Nov 03 23:02:23.434009 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.433991 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.451067 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.451032 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4"] Nov 03 23:02:23.455506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.454444 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:23.455506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.454475 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:02:23.455506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.454529 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:02:23.457325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.457304 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:02:23.558395 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.558352 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86lp\" (UniqueName: \"kubernetes.io/projected/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-kube-api-access-z86lp\") pod \"maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.558582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.558411 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.558582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.558539 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.660282 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.660247 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86lp\" (UniqueName: \"kubernetes.io/projected/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-kube-api-access-z86lp\") pod \"maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.660474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.660305 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.660474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.660340 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.661090 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.661065 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.662743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.662724 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.683582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.683515 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86lp\" (UniqueName: \"kubernetes.io/projected/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-kube-api-access-z86lp\") pod \"maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:23.752144 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:23.752101 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:24.074705 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:02:24.074669 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2ccb9f_bd02_4ce2_b3ba_3f24c146335f.slice/crio-3630bb462eb050511b32c0d95371174d54eb2f88f227e3a4343f727acd6d79d6 WatchSource:0}: Error finding container 3630bb462eb050511b32c0d95371174d54eb2f88f227e3a4343f727acd6d79d6: Status 404 returned error can't find the container with id 3630bb462eb050511b32c0d95371174d54eb2f88f227e3a4343f727acd6d79d6 Nov 03 23:02:24.075026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:24.075004 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4"] Nov 03 23:02:24.803346 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:24.803286 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" event={"ID":"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f","Type":"ContainerStarted","Data":"738d2823840ccc9014d2513d01c0b6692b2a74470568d4815c45b21a015e8f17"} Nov 03 23:02:24.803741 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:24.803372 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" event={"ID":"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f","Type":"ContainerStarted","Data":"3630bb462eb050511b32c0d95371174d54eb2f88f227e3a4343f727acd6d79d6"} Nov 03 23:02:24.840779 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:24.840714 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" podStartSLOduration=1.840694716 podStartE2EDuration="1.840694716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:24.838997793 +0000 UTC m=+2971.242938769" watchObservedRunningTime="2026-11-03 23:02:24.840694716 +0000 UTC m=+2971.244635691" Nov 03 23:02:25.818603 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:25.818570 4903 generic.go:356] "Generic (PLEG): container finished" podID="cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f" containerID="738d2823840ccc9014d2513d01c0b6692b2a74470568d4815c45b21a015e8f17" exitCode=0 Nov 03 23:02:25.819034 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:25.818660 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" event={"ID":"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f","Type":"ContainerDied","Data":"738d2823840ccc9014d2513d01c0b6692b2a74470568d4815c45b21a015e8f17"} Nov 03 23:02:26.035058 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:26.035019 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd"] Nov 03 23:02:26.041102 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:26.041068 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e687d8-f9zhd"] Nov 03 23:02:26.984710 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:26.984682 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:27.135368 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.135255 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-xks-ca\") pod \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " Nov 03 23:02:27.135532 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.135514 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-openshift-ca\") pod \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " Nov 03 23:02:27.135657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.135640 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-kube-api-access-z86lp\" (UniqueName: \"kubernetes.io/projected/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-kube-api-access-z86lp\") pod \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\" (UID: \"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f\") " Nov 03 23:02:27.135876 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.135853 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-openshift-ca" pod "cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f" (UID: "cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:27.136101 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.136082 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:27.137482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.137463 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-xks-ca" pod "cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f" (UID: "cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:27.137731 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.137707 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-kube-api-access-z86lp" pod "cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f" (UID: "cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f"). InnerVolumeSpecName "kube-api-access-z86lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:27.238012 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.237979 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z86lp\" (UniqueName: \"kubernetes.io/projected/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-kube-api-access-z86lp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:27.238012 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.238009 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:27.735924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.735889 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6" path="/var/lib/kubelet/pods/fea8c72e-7ae7-4d64-a5db-fa6bb2e9a9b6/volumes" Nov 03 23:02:27.839966 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.839939 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" Nov 03 23:02:27.840136 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.840024 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4" event={"ID":"cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f","Type":"ContainerDied","Data":"3630bb462eb050511b32c0d95371174d54eb2f88f227e3a4343f727acd6d79d6"} Nov 03 23:02:27.840136 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:27.840070 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3630bb462eb050511b32c0d95371174d54eb2f88f227e3a4343f727acd6d79d6" Nov 03 23:02:28.756198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:28.756168 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:02:36.030249 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:36.030210 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5"] Nov 03 23:02:36.039401 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:36.039375 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd824a5-mdcl5"] Nov 03 23:02:36.746099 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:36.746068 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw"] Nov 03 23:02:36.755726 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:36.755692 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e19850-7znrw"] Nov 03 23:02:37.736041 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:37.736001 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0cd899-de8f-4ff4-bf9b-03fc5a132abe" path="/var/lib/kubelet/pods/4a0cd899-de8f-4ff4-bf9b-03fc5a132abe/volumes" Nov 03 23:02:37.736694 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:37.736673 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67e4b26-0628-42c0-84cf-dd4d86637b49" path="/var/lib/kubelet/pods/f67e4b26-0628-42c0-84cf-dd4d86637b49/volumes" Nov 03 23:02:38.177706 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.177674 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2"] Nov 03 23:02:38.178437 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.178422 4903 cpu_manager.go:414] "containerMap: removing container" podUID="cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f" containerName="revoke-keys" Nov 03 23:02:38.178517 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.178438 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f" containerName="revoke-keys" Nov 03 23:02:38.178643 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.178627 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f" containerName="revoke-keys" Nov 03 23:02:38.184503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.184481 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.200866 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.200840 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2"] Nov 03 23:02:38.205996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.205968 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:38.205996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.205978 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:02:38.206174 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.206000 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:02:38.208778 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.208758 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:02:38.289123 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.289086 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-856xm\" (UniqueName: \"kubernetes.io/projected/791668bf-7e0e-4569-8e40-f0a87c2d6b55-kube-api-access-856xm\") pod \"maas-api-revoke-sub-b9456c75-wvgf2\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.289320 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.289164 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9456c75-wvgf2\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.289389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.289361 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9456c75-wvgf2\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.391184 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.391145 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9456c75-wvgf2\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.391424 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.391273 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-856xm\" (UniqueName: \"kubernetes.io/projected/791668bf-7e0e-4569-8e40-f0a87c2d6b55-kube-api-access-856xm\") pod \"maas-api-revoke-sub-b9456c75-wvgf2\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.391424 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.391378 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9456c75-wvgf2\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.391879 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.391856 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9456c75-wvgf2\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.394092 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.394067 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9456c75-wvgf2\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.415333 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.415278 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-856xm\" (UniqueName: \"kubernetes.io/projected/791668bf-7e0e-4569-8e40-f0a87c2d6b55-kube-api-access-856xm\") pod \"maas-api-revoke-sub-b9456c75-wvgf2\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.503431 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.503356 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:38.846717 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.846640 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2"] Nov 03 23:02:38.856726 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:02:38.856696 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791668bf_7e0e_4569_8e40_f0a87c2d6b55.slice/crio-7db2db14f6479659202fcc90d0ec6922ebc2d6cdc1a25966f32d67850ea03bfa WatchSource:0}: Error finding container 7db2db14f6479659202fcc90d0ec6922ebc2d6cdc1a25966f32d67850ea03bfa: Status 404 returned error can't find the container with id 7db2db14f6479659202fcc90d0ec6922ebc2d6cdc1a25966f32d67850ea03bfa Nov 03 23:02:38.935665 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:38.935632 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" event={"ID":"791668bf-7e0e-4569-8e40-f0a87c2d6b55","Type":"ContainerStarted","Data":"7db2db14f6479659202fcc90d0ec6922ebc2d6cdc1a25966f32d67850ea03bfa"} Nov 03 23:02:39.944445 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:39.944402 4903 generic.go:356] "Generic (PLEG): container finished" podID="791668bf-7e0e-4569-8e40-f0a87c2d6b55" containerID="3bf0ed812a1c5f3430064cbadb285a86a8d37a820eb0978ede838d89e64ec9f4" exitCode=0 Nov 03 23:02:39.944850 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:39.944488 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" event={"ID":"791668bf-7e0e-4569-8e40-f0a87c2d6b55","Type":"ContainerDied","Data":"3bf0ed812a1c5f3430064cbadb285a86a8d37a820eb0978ede838d89e64ec9f4"} Nov 03 23:02:41.119890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.119865 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:41.228868 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.228781 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-openshift-ca\") pod \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " Nov 03 23:02:41.229033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.228875 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/791668bf-7e0e-4569-8e40-f0a87c2d6b55-kube-api-access-856xm\" (UniqueName: \"kubernetes.io/projected/791668bf-7e0e-4569-8e40-f0a87c2d6b55-kube-api-access-856xm\") pod \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " Nov 03 23:02:41.229033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.228973 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-xks-ca\") pod \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\" (UID: \"791668bf-7e0e-4569-8e40-f0a87c2d6b55\") " Nov 03 23:02:41.229221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.229200 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-openshift-ca" pod "791668bf-7e0e-4569-8e40-f0a87c2d6b55" (UID: "791668bf-7e0e-4569-8e40-f0a87c2d6b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:41.229561 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.229543 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:41.231057 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.231030 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-xks-ca" pod "791668bf-7e0e-4569-8e40-f0a87c2d6b55" (UID: "791668bf-7e0e-4569-8e40-f0a87c2d6b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:41.231182 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.231087 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791668bf-7e0e-4569-8e40-f0a87c2d6b55-kube-api-access-856xm" pod "791668bf-7e0e-4569-8e40-f0a87c2d6b55" (UID: "791668bf-7e0e-4569-8e40-f0a87c2d6b55"). InnerVolumeSpecName "kube-api-access-856xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:41.330874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.330841 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-856xm\" (UniqueName: \"kubernetes.io/projected/791668bf-7e0e-4569-8e40-f0a87c2d6b55-kube-api-access-856xm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:41.330874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.330869 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791668bf-7e0e-4569-8e40-f0a87c2d6b55-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:41.961952 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.961920 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" Nov 03 23:02:41.962113 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.961956 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2" event={"ID":"791668bf-7e0e-4569-8e40-f0a87c2d6b55","Type":"ContainerDied","Data":"7db2db14f6479659202fcc90d0ec6922ebc2d6cdc1a25966f32d67850ea03bfa"} Nov 03 23:02:41.962113 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:41.962007 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7db2db14f6479659202fcc90d0ec6922ebc2d6cdc1a25966f32d67850ea03bfa" Nov 03 23:02:42.470739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:42.470706 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w"] Nov 03 23:02:42.474862 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:42.474834 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a01f9-xhf2w"] Nov 03 23:02:43.736616 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:43.736586 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba195c6-6516-463d-a553-45081c6b7c71" path="/var/lib/kubelet/pods/cba195c6-6516-463d-a553-45081c6b7c71/volumes" Nov 03 23:02:48.384953 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:02:48.384914 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:02:49.742774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.742733 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9"] Nov 03 23:02:49.743550 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.743530 4903 cpu_manager.go:414] "containerMap: removing container" podUID="791668bf-7e0e-4569-8e40-f0a87c2d6b55" containerName="revoke-keys" Nov 03 23:02:49.743550 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.743550 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="791668bf-7e0e-4569-8e40-f0a87c2d6b55" containerName="revoke-keys" Nov 03 23:02:49.743710 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.743699 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="791668bf-7e0e-4569-8e40-f0a87c2d6b55" containerName="revoke-keys" Nov 03 23:02:49.748851 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.748833 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:49.763999 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.763972 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9"] Nov 03 23:02:49.767729 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.767703 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:02:49.767834 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.767750 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:02:49.791806 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.791774 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:02:49.791959 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.791839 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:02:49.865275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.865242 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:49.865536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.865312 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5mm\" (UniqueName: \"kubernetes.io/projected/d256e8fe-9234-4879-9126-7c869899ea70-kube-api-access-pr5mm\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:49.865536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.865400 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:49.967393 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.967356 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:49.967567 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.967405 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5mm\" (UniqueName: \"kubernetes.io/projected/d256e8fe-9234-4879-9126-7c869899ea70-kube-api-access-pr5mm\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:49.967567 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.967450 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:49.968116 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.968086 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:49.969808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.969784 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:49.988981 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:49.988958 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5mm\" (UniqueName: \"kubernetes.io/projected/d256e8fe-9234-4879-9126-7c869899ea70-kube-api-access-pr5mm\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:50.067382 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:50.067265 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:50.376098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:50.376067 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9"] Nov 03 23:02:50.376464 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:02:50.376440 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd256e8fe_9234_4879_9126_7c869899ea70.slice/crio-a0ada38d0d822c38484217a0c518bb5de98696a62a132c771706643e3f7320cb WatchSource:0}: Error finding container a0ada38d0d822c38484217a0c518bb5de98696a62a132c771706643e3f7320cb: Status 404 returned error can't find the container with id a0ada38d0d822c38484217a0c518bb5de98696a62a132c771706643e3f7320cb Nov 03 23:02:51.050243 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:51.050153 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" event={"ID":"d256e8fe-9234-4879-9126-7c869899ea70","Type":"ContainerStarted","Data":"5d48370d7ff06aacd96a24f5b0f9f27bdfee87b8f90d26c15a135370b1d245cc"} Nov 03 23:02:51.050243 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:51.050208 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" event={"ID":"d256e8fe-9234-4879-9126-7c869899ea70","Type":"ContainerStarted","Data":"a0ada38d0d822c38484217a0c518bb5de98696a62a132c771706643e3f7320cb"} Nov 03 23:02:51.088982 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:51.088928 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" podStartSLOduration=2.088915717 podStartE2EDuration="2.088915717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:02:51.087019861 +0000 UTC m=+2997.490960835" watchObservedRunningTime="2026-11-03 23:02:51.088915717 +0000 UTC m=+2997.492856692" Nov 03 23:02:52.060642 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:52.060608 4903 generic.go:356] "Generic (PLEG): container finished" podID="d256e8fe-9234-4879-9126-7c869899ea70" containerID="5d48370d7ff06aacd96a24f5b0f9f27bdfee87b8f90d26c15a135370b1d245cc" exitCode=0 Nov 03 23:02:52.061076 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:52.060689 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" event={"ID":"d256e8fe-9234-4879-9126-7c869899ea70","Type":"ContainerDied","Data":"5d48370d7ff06aacd96a24f5b0f9f27bdfee87b8f90d26c15a135370b1d245cc"} Nov 03 23:02:53.264679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.264651 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:53.315538 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.315507 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-xks-ca\") pod \"d256e8fe-9234-4879-9126-7c869899ea70\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " Nov 03 23:02:53.315742 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.315704 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d256e8fe-9234-4879-9126-7c869899ea70-kube-api-access-pr5mm\" (UniqueName: \"kubernetes.io/projected/d256e8fe-9234-4879-9126-7c869899ea70-kube-api-access-pr5mm\") pod \"d256e8fe-9234-4879-9126-7c869899ea70\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " Nov 03 23:02:53.315823 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.315794 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-openshift-ca\") pod \"d256e8fe-9234-4879-9126-7c869899ea70\" (UID: \"d256e8fe-9234-4879-9126-7c869899ea70\") " Nov 03 23:02:53.316225 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.316191 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-openshift-ca" pod "d256e8fe-9234-4879-9126-7c869899ea70" (UID: "d256e8fe-9234-4879-9126-7c869899ea70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:02:53.316458 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.316434 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:53.317853 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.317821 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-xks-ca" pod "d256e8fe-9234-4879-9126-7c869899ea70" (UID: "d256e8fe-9234-4879-9126-7c869899ea70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:02:53.318382 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.318357 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d256e8fe-9234-4879-9126-7c869899ea70-kube-api-access-pr5mm" pod "d256e8fe-9234-4879-9126-7c869899ea70" (UID: "d256e8fe-9234-4879-9126-7c869899ea70"). InnerVolumeSpecName "kube-api-access-pr5mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:02:53.417343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.417311 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr5mm\" (UniqueName: \"kubernetes.io/projected/d256e8fe-9234-4879-9126-7c869899ea70-kube-api-access-pr5mm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:53.417343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:53.417340 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d256e8fe-9234-4879-9126-7c869899ea70-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:02:54.082777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:54.082694 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" Nov 03 23:02:54.082777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:54.082725 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9" event={"ID":"d256e8fe-9234-4879-9126-7c869899ea70","Type":"ContainerDied","Data":"a0ada38d0d822c38484217a0c518bb5de98696a62a132c771706643e3f7320cb"} Nov 03 23:02:54.082777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:54.082769 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0ada38d0d822c38484217a0c518bb5de98696a62a132c771706643e3f7320cb" Nov 03 23:02:54.380608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:54.380528 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:02:54.388372 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:54.388347 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:02:54.388544 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:54.388456 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:02:54.395853 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:54.395834 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:02:58.874753 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:58.874720 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7"] Nov 03 23:02:58.905365 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:02:58.905332 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs"] Nov 03 23:03:00.381231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:00.381141 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc"] Nov 03 23:03:01.061603 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:01.061566 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:03:03.875564 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:03.875513 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" podUID="aad1242d-05c2-4478-84fb-d3fe522001a3" containerName="payload-processing" containerID="cri-o://437bb3ce731edc21692aff2fa08af27d1ce29d9c5cfbc65279290ad5e184c386" gracePeriod=25 Nov 03 23:03:03.906600 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:03.906564 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" podUID="55c3f9f2-b9ee-4605-9854-2e5778c4ad9c" containerName="payload-pre-processing" containerID="cri-o://39c07874915c2ae97372f0df20fd3089493ca770ad15164a0ac35eeab1614a9a" gracePeriod=25 Nov 03 23:03:04.228841 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.228805 4903 generic.go:356] "Generic (PLEG): container finished" podID="55c3f9f2-b9ee-4605-9854-2e5778c4ad9c" containerID="39c07874915c2ae97372f0df20fd3089493ca770ad15164a0ac35eeab1614a9a" exitCode=0 Nov 03 23:03:04.229101 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.228937 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" event={"ID":"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c","Type":"ContainerDied","Data":"39c07874915c2ae97372f0df20fd3089493ca770ad15164a0ac35eeab1614a9a"} Nov 03 23:03:04.238466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.238420 4903 generic.go:356] "Generic (PLEG): container finished" podID="aad1242d-05c2-4478-84fb-d3fe522001a3" containerID="437bb3ce731edc21692aff2fa08af27d1ce29d9c5cfbc65279290ad5e184c386" exitCode=0 Nov 03 23:03:04.238709 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.238692 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" event={"ID":"aad1242d-05c2-4478-84fb-d3fe522001a3","Type":"ContainerDied","Data":"437bb3ce731edc21692aff2fa08af27d1ce29d9c5cfbc65279290ad5e184c386"} Nov 03 23:03:04.301731 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.301695 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:03:04.338073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.338039 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:03:04.401691 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.401646 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-kube-api-access-knsnm\" (UniqueName: \"kubernetes.io/projected/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-kube-api-access-knsnm\") pod \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\" (UID: \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\") " Nov 03 23:03:04.401888 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.401713 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-plugins-config-volume\") pod \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\" (UID: \"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c\") " Nov 03 23:03:04.401952 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.401925 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad1242d-05c2-4478-84fb-d3fe522001a3-kube-api-access-8h474\" (UniqueName: \"kubernetes.io/projected/aad1242d-05c2-4478-84fb-d3fe522001a3-kube-api-access-8h474\") pod \"aad1242d-05c2-4478-84fb-d3fe522001a3\" (UID: \"aad1242d-05c2-4478-84fb-d3fe522001a3\") " Nov 03 23:03:04.402013 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.401964 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad1242d-05c2-4478-84fb-d3fe522001a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad1242d-05c2-4478-84fb-d3fe522001a3-plugins-config-volume\") pod \"aad1242d-05c2-4478-84fb-d3fe522001a3\" (UID: \"aad1242d-05c2-4478-84fb-d3fe522001a3\") " Nov 03 23:03:04.402955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.402925 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad1242d-05c2-4478-84fb-d3fe522001a3-plugins-config-volume" pod "aad1242d-05c2-4478-84fb-d3fe522001a3" (UID: "aad1242d-05c2-4478-84fb-d3fe522001a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:04.403281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.403252 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-plugins-config-volume" pod "55c3f9f2-b9ee-4605-9854-2e5778c4ad9c" (UID: "55c3f9f2-b9ee-4605-9854-2e5778c4ad9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:04.405057 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.405034 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad1242d-05c2-4478-84fb-d3fe522001a3-kube-api-access-8h474" pod "aad1242d-05c2-4478-84fb-d3fe522001a3" (UID: "aad1242d-05c2-4478-84fb-d3fe522001a3"). InnerVolumeSpecName "kube-api-access-8h474". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:04.407253 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.407226 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-kube-api-access-knsnm" pod "55c3f9f2-b9ee-4605-9854-2e5778c4ad9c" (UID: "55c3f9f2-b9ee-4605-9854-2e5778c4ad9c"). InnerVolumeSpecName "kube-api-access-knsnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:04.503398 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.503312 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knsnm\" (UniqueName: \"kubernetes.io/projected/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-kube-api-access-knsnm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:04.503398 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.503346 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:04.503398 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.503355 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h474\" (UniqueName: \"kubernetes.io/projected/aad1242d-05c2-4478-84fb-d3fe522001a3-kube-api-access-8h474\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:04.503398 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:04.503364 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad1242d-05c2-4478-84fb-d3fe522001a3-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:05.268709 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.268456 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" Nov 03 23:03:05.268709 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.268507 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs" event={"ID":"55c3f9f2-b9ee-4605-9854-2e5778c4ad9c","Type":"ContainerDied","Data":"a6eea19ef15cf8db9364548559678debc932b2e57ab977e67a991052eb9fd144"} Nov 03 23:03:05.268709 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.268590 4903 scope.go:122] "RemoveContainer" containerID="39c07874915c2ae97372f0df20fd3089493ca770ad15164a0ac35eeab1614a9a" Nov 03 23:03:05.270773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.270753 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" Nov 03 23:03:05.270917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.270894 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7" event={"ID":"aad1242d-05c2-4478-84fb-d3fe522001a3","Type":"ContainerDied","Data":"6345b2b3168588300d25b8d30317682b9c8d283784c71f9aad3b8cd3bf6a33b0"} Nov 03 23:03:05.283562 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.283542 4903 scope.go:122] "RemoveContainer" containerID="437bb3ce731edc21692aff2fa08af27d1ce29d9c5cfbc65279290ad5e184c386" Nov 03 23:03:05.317559 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.317527 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7"] Nov 03 23:03:05.321562 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.321538 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ad693-5cb9c56db6-kvcc7"] Nov 03 23:03:05.370646 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.370610 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs"] Nov 03 23:03:05.374685 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.374659 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ad693-5cfb5979-pq6rs"] Nov 03 23:03:05.382080 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.382037 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" podUID="17e621b5-3a21-4a9d-b192-dc3c126578a6" containerName="payload-pre-processing" containerID="cri-o://5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0" gracePeriod=25 Nov 03 23:03:05.711944 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.711921 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 23:03:05.720252 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.720219 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e621b5-3a21-4a9d-b192-dc3c126578a6-kube-api-access-bc6f5\" (UniqueName: \"kubernetes.io/projected/17e621b5-3a21-4a9d-b192-dc3c126578a6-kube-api-access-bc6f5\") pod \"17e621b5-3a21-4a9d-b192-dc3c126578a6\" (UID: \"17e621b5-3a21-4a9d-b192-dc3c126578a6\") " Nov 03 23:03:05.720503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.720482 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e621b5-3a21-4a9d-b192-dc3c126578a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e621b5-3a21-4a9d-b192-dc3c126578a6-plugins-config-volume\") pod \"17e621b5-3a21-4a9d-b192-dc3c126578a6\" (UID: \"17e621b5-3a21-4a9d-b192-dc3c126578a6\") " Nov 03 23:03:05.720830 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.720809 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e621b5-3a21-4a9d-b192-dc3c126578a6-plugins-config-volume" pod "17e621b5-3a21-4a9d-b192-dc3c126578a6" (UID: "17e621b5-3a21-4a9d-b192-dc3c126578a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:05.721248 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.721231 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e621b5-3a21-4a9d-b192-dc3c126578a6-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:05.722343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.722326 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e621b5-3a21-4a9d-b192-dc3c126578a6-kube-api-access-bc6f5" pod "17e621b5-3a21-4a9d-b192-dc3c126578a6" (UID: "17e621b5-3a21-4a9d-b192-dc3c126578a6"). InnerVolumeSpecName "kube-api-access-bc6f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:05.735804 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.735779 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c3f9f2-b9ee-4605-9854-2e5778c4ad9c" path="/var/lib/kubelet/pods/55c3f9f2-b9ee-4605-9854-2e5778c4ad9c/volumes" Nov 03 23:03:05.736283 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.736263 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad1242d-05c2-4478-84fb-d3fe522001a3" path="/var/lib/kubelet/pods/aad1242d-05c2-4478-84fb-d3fe522001a3/volumes" Nov 03 23:03:05.822617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:05.822579 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc6f5\" (UniqueName: \"kubernetes.io/projected/17e621b5-3a21-4a9d-b192-dc3c126578a6-kube-api-access-bc6f5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:06.281864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:06.281767 4903 generic.go:356] "Generic (PLEG): container finished" podID="17e621b5-3a21-4a9d-b192-dc3c126578a6" containerID="5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0" exitCode=0 Nov 03 23:03:06.281864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:06.281817 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" Nov 03 23:03:06.281864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:06.281851 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" event={"ID":"17e621b5-3a21-4a9d-b192-dc3c126578a6","Type":"ContainerDied","Data":"5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0"} Nov 03 23:03:06.282449 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:06.281931 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc" event={"ID":"17e621b5-3a21-4a9d-b192-dc3c126578a6","Type":"ContainerDied","Data":"0bba0044f7507e593343ffb6c6377562d41561f6ca53112fc378a56d67f42782"} Nov 03 23:03:06.282449 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:06.281976 4903 scope.go:122] "RemoveContainer" containerID="5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0" Nov 03 23:03:06.296732 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:06.296705 4903 scope.go:122] "RemoveContainer" containerID="5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0" Nov 03 23:03:06.297127 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:03:06.297099 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0\": container with ID starting with 5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0 not found: ID does not exist" containerID="5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0" Nov 03 23:03:06.297243 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:06.297136 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0"} err="failed to get container status \"5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0\": rpc error: code = NotFound desc = could not find container \"5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0\": container with ID starting with 5f21293791d1ec76b8b659f3a7e40d773ce6d49a34880f2e07f5efdd42bf1ca0 not found: ID does not exist" Nov 03 23:03:06.336167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:06.336139 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc"] Nov 03 23:03:06.339984 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:06.339959 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d4b6bb-7678b5f4f9-d68cc"] Nov 03 23:03:07.596165 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:07.596129 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh"] Nov 03 23:03:07.627534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:07.627501 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp"] Nov 03 23:03:07.744271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:07.744235 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e621b5-3a21-4a9d-b192-dc3c126578a6" path="/var/lib/kubelet/pods/17e621b5-3a21-4a9d-b192-dc3c126578a6/volumes" Nov 03 23:03:10.400452 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:10.400412 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:03:10.400924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:10.400473 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:03:12.352032 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.351998 4903 generic.go:356] "Generic (PLEG): container finished" podID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerID="e603f5b193260da999995423ea21b8587b309ab0cff2d08b3afe86e10ad26a75" exitCode=0 Nov 03 23:03:12.352635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.352072 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" event={"ID":"2ea5c865-19c0-4343-b2d5-f34a94171fa8","Type":"ContainerDied","Data":"e603f5b193260da999995423ea21b8587b309ab0cff2d08b3afe86e10ad26a75"} Nov 03 23:03:12.352718 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.352632 4903 scope.go:122] "RemoveContainer" containerID="e603f5b193260da999995423ea21b8587b309ab0cff2d08b3afe86e10ad26a75" Nov 03 23:03:12.354454 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.354429 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:03:12.354526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.354496 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:03:12.354573 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.354529 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:03:12.597056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.596966 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" podUID="0b50a9c0-0dc4-4b45-9ce4-efc3182d7450" containerName="payload-processing" containerID="cri-o://2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0" gracePeriod=25 Nov 03 23:03:12.628752 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.628708 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" podUID="4dfb70f9-51cc-4249-b088-af813d7b54dc" containerName="payload-pre-processing" containerID="cri-o://8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35" gracePeriod=25 Nov 03 23:03:12.957400 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.957378 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:03:12.960928 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:12.960910 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:03:13.047045 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.047007 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfb70f9-51cc-4249-b088-af813d7b54dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfb70f9-51cc-4249-b088-af813d7b54dc-plugins-config-volume\") pod \"4dfb70f9-51cc-4249-b088-af813d7b54dc\" (UID: \"4dfb70f9-51cc-4249-b088-af813d7b54dc\") " Nov 03 23:03:13.047238 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.047064 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfb70f9-51cc-4249-b088-af813d7b54dc-kube-api-access-cf8p6\" (UniqueName: \"kubernetes.io/projected/4dfb70f9-51cc-4249-b088-af813d7b54dc-kube-api-access-cf8p6\") pod \"4dfb70f9-51cc-4249-b088-af813d7b54dc\" (UID: \"4dfb70f9-51cc-4249-b088-af813d7b54dc\") " Nov 03 23:03:13.047238 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.047115 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-plugins-config-volume\") pod \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\" (UID: \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\") " Nov 03 23:03:13.047417 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.047251 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-kube-api-access-5lkbh\" (UniqueName: \"kubernetes.io/projected/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-kube-api-access-5lkbh\") pod \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\" (UID: \"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450\") " Nov 03 23:03:13.048237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.047503 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfb70f9-51cc-4249-b088-af813d7b54dc-plugins-config-volume" pod "4dfb70f9-51cc-4249-b088-af813d7b54dc" (UID: "4dfb70f9-51cc-4249-b088-af813d7b54dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:13.048237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.047646 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-plugins-config-volume" pod "0b50a9c0-0dc4-4b45-9ce4-efc3182d7450" (UID: "0b50a9c0-0dc4-4b45-9ce4-efc3182d7450"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:13.048237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.048101 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dfb70f9-51cc-4249-b088-af813d7b54dc-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:13.048237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.048122 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:13.049993 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.049966 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfb70f9-51cc-4249-b088-af813d7b54dc-kube-api-access-cf8p6" pod "4dfb70f9-51cc-4249-b088-af813d7b54dc" (UID: "4dfb70f9-51cc-4249-b088-af813d7b54dc"). InnerVolumeSpecName "kube-api-access-cf8p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:13.050435 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.050416 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-kube-api-access-5lkbh" pod "0b50a9c0-0dc4-4b45-9ce4-efc3182d7450" (UID: "0b50a9c0-0dc4-4b45-9ce4-efc3182d7450"). InnerVolumeSpecName "kube-api-access-5lkbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:13.149256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.149220 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lkbh\" (UniqueName: \"kubernetes.io/projected/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450-kube-api-access-5lkbh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:13.149256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.149253 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf8p6\" (UniqueName: \"kubernetes.io/projected/4dfb70f9-51cc-4249-b088-af813d7b54dc-kube-api-access-cf8p6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:13.377220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.377140 4903 generic.go:356] "Generic (PLEG): container finished" podID="4dfb70f9-51cc-4249-b088-af813d7b54dc" containerID="8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35" exitCode=0 Nov 03 23:03:13.377220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.377184 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" Nov 03 23:03:13.377220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.377204 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" event={"ID":"4dfb70f9-51cc-4249-b088-af813d7b54dc","Type":"ContainerDied","Data":"8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35"} Nov 03 23:03:13.377818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.377258 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp" event={"ID":"4dfb70f9-51cc-4249-b088-af813d7b54dc","Type":"ContainerDied","Data":"071a8e70fe09e8b2b23bc3b5d7312026c7a6ceec6bd900899d30bb8108c83356"} Nov 03 23:03:13.377818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.377284 4903 scope.go:122] "RemoveContainer" containerID="8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35" Nov 03 23:03:13.379836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.379810 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" event={"ID":"2ea5c865-19c0-4343-b2d5-f34a94171fa8","Type":"ContainerStarted","Data":"0b81b9e163ebdc4205b95c923e53e574116cabd8fc9cb500b44d0749bc98822a"} Nov 03 23:03:13.381532 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.381512 4903 generic.go:356] "Generic (PLEG): container finished" podID="0b50a9c0-0dc4-4b45-9ce4-efc3182d7450" containerID="2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0" exitCode=0 Nov 03 23:03:13.381635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.381563 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" Nov 03 23:03:13.381635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.381572 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" event={"ID":"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450","Type":"ContainerDied","Data":"2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0"} Nov 03 23:03:13.381635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.381620 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh" event={"ID":"0b50a9c0-0dc4-4b45-9ce4-efc3182d7450","Type":"ContainerDied","Data":"1a1da6bda4d8af663be63ec6f36ceef6d5efebe24af6d497fa6e1a7e5f49021e"} Nov 03 23:03:13.391880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.391864 4903 scope.go:122] "RemoveContainer" containerID="8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35" Nov 03 23:03:13.392164 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:03:13.392140 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35\": container with ID starting with 8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35 not found: ID does not exist" containerID="8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35" Nov 03 23:03:13.392255 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.392169 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35"} err="failed to get container status \"8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35\": rpc error: code = NotFound desc = could not find container \"8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35\": container with ID starting with 8a9201ea17612a62a591c49e92ab8710c3e5a6853733c85e5dff1cb97d349a35 not found: ID does not exist" Nov 03 23:03:13.392255 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.392184 4903 scope.go:122] "RemoveContainer" containerID="2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0" Nov 03 23:03:13.402216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.402199 4903 scope.go:122] "RemoveContainer" containerID="2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0" Nov 03 23:03:13.402495 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:03:13.402478 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0\": container with ID starting with 2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0 not found: ID does not exist" containerID="2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0" Nov 03 23:03:13.402555 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.402501 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0"} err="failed to get container status \"2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0\": rpc error: code = NotFound desc = could not find container \"2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0\": container with ID starting with 2d9492fc9c9a33dbe9656207acee56e6e9b633f204a94cba71e737812e8b22d0 not found: ID does not exist" Nov 03 23:03:13.465383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.463924 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:03:13.465383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.464483 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 23:03:13.465383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.464528 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 23:03:13.500651 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.500618 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp"] Nov 03 23:03:13.506769 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.506739 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-162d7b-c8d776ff9-cswcp"] Nov 03 23:03:13.538359 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.538334 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh"] Nov 03 23:03:13.568179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.568133 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-162d7b-d6c56f5db-nggjh"] Nov 03 23:03:13.773342 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.773288 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b50a9c0-0dc4-4b45-9ce4-efc3182d7450" path="/var/lib/kubelet/pods/0b50a9c0-0dc4-4b45-9ce4-efc3182d7450/volumes" Nov 03 23:03:13.774118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:13.773976 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfb70f9-51cc-4249-b088-af813d7b54dc" path="/var/lib/kubelet/pods/4dfb70f9-51cc-4249-b088-af813d7b54dc/volumes" Nov 03 23:03:14.465317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:14.464801 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 23:03:14.465317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:14.464872 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 23:03:15.466087 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:15.465455 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 23:03:15.466087 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:15.465546 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 23:03:16.459355 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:16.459319 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 23:03:16.460583 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:16.460551 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 23:03:17.466318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:17.462621 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 23:03:17.466318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:17.462681 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 23:03:18.172242 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.172100 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2"] Nov 03 23:03:18.172489 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.172424 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" podUID="53bd4dbd-7d51-4454-a43b-b8f5680c3857" containerName="payload-processing" containerID="cri-o://9b5eb9b6995b80769fab1f89cc42f8dcef1fcb13ba5d0077138d04023432d52d" gracePeriod=30 Nov 03 23:03:18.184858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.176802 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5"] Nov 03 23:03:18.184858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.177380 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" podUID="f210a637-436f-459e-9b41-fe49955a0e3c" containerName="payload-pre-processing" containerID="cri-o://63438a1f51cfd71ae008bdc5bc25e03115edb624c8d47f632623492278957434" gracePeriod=30 Nov 03 23:03:18.404600 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:03:18.404565 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:03:18.484386 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.472453 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 23:03:18.495806 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.495762 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 23:03:18.580314 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.579530 4903 generic.go:356] "Generic (PLEG): container finished" podID="53bd4dbd-7d51-4454-a43b-b8f5680c3857" containerID="9b5eb9b6995b80769fab1f89cc42f8dcef1fcb13ba5d0077138d04023432d52d" exitCode=0 Nov 03 23:03:18.580314 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.580010 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" event={"ID":"53bd4dbd-7d51-4454-a43b-b8f5680c3857","Type":"ContainerDied","Data":"9b5eb9b6995b80769fab1f89cc42f8dcef1fcb13ba5d0077138d04023432d52d"} Nov 03 23:03:18.596754 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.596717 4903 generic.go:356] "Generic (PLEG): container finished" podID="f210a637-436f-459e-9b41-fe49955a0e3c" containerID="63438a1f51cfd71ae008bdc5bc25e03115edb624c8d47f632623492278957434" exitCode=0 Nov 03 23:03:18.597170 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.597100 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" event={"ID":"f210a637-436f-459e-9b41-fe49955a0e3c","Type":"ContainerDied","Data":"63438a1f51cfd71ae008bdc5bc25e03115edb624c8d47f632623492278957434"} Nov 03 23:03:18.661118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.658942 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 23:03:18.701736 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.701695 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53bd4dbd-7d51-4454-a43b-b8f5680c3857-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bd4dbd-7d51-4454-a43b-b8f5680c3857-plugins-config-volume\") pod \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\" (UID: \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\") " Nov 03 23:03:18.701917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.701807 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53bd4dbd-7d51-4454-a43b-b8f5680c3857-kube-api-access-xb96h\" (UniqueName: \"kubernetes.io/projected/53bd4dbd-7d51-4454-a43b-b8f5680c3857-kube-api-access-xb96h\") pod \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\" (UID: \"53bd4dbd-7d51-4454-a43b-b8f5680c3857\") " Nov 03 23:03:18.703748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.703717 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bd4dbd-7d51-4454-a43b-b8f5680c3857-plugins-config-volume" pod "53bd4dbd-7d51-4454-a43b-b8f5680c3857" (UID: "53bd4dbd-7d51-4454-a43b-b8f5680c3857"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:18.705860 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.705827 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bd4dbd-7d51-4454-a43b-b8f5680c3857-kube-api-access-xb96h" pod "53bd4dbd-7d51-4454-a43b-b8f5680c3857" (UID: "53bd4dbd-7d51-4454-a43b-b8f5680c3857"). InnerVolumeSpecName "kube-api-access-xb96h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:18.710025 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.710005 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 23:03:18.788176 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.788089 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4"] Nov 03 23:03:18.792749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.792724 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3920cb-znhb4"] Nov 03 23:03:18.803244 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.803218 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f210a637-436f-459e-9b41-fe49955a0e3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f210a637-436f-459e-9b41-fe49955a0e3c-plugins-config-volume\") pod \"f210a637-436f-459e-9b41-fe49955a0e3c\" (UID: \"f210a637-436f-459e-9b41-fe49955a0e3c\") " Nov 03 23:03:18.803403 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.803383 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f210a637-436f-459e-9b41-fe49955a0e3c-kube-api-access-hwklp\" (UniqueName: \"kubernetes.io/projected/f210a637-436f-459e-9b41-fe49955a0e3c-kube-api-access-hwklp\") pod \"f210a637-436f-459e-9b41-fe49955a0e3c\" (UID: \"f210a637-436f-459e-9b41-fe49955a0e3c\") " Nov 03 23:03:18.803674 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.803645 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f210a637-436f-459e-9b41-fe49955a0e3c-plugins-config-volume" pod "f210a637-436f-459e-9b41-fe49955a0e3c" (UID: "f210a637-436f-459e-9b41-fe49955a0e3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:18.804014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.803991 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53bd4dbd-7d51-4454-a43b-b8f5680c3857-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:18.804014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.804015 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f210a637-436f-459e-9b41-fe49955a0e3c-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:18.804192 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.804024 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb96h\" (UniqueName: \"kubernetes.io/projected/53bd4dbd-7d51-4454-a43b-b8f5680c3857-kube-api-access-xb96h\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:18.806133 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.806114 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f210a637-436f-459e-9b41-fe49955a0e3c-kube-api-access-hwklp" pod "f210a637-436f-459e-9b41-fe49955a0e3c" (UID: "f210a637-436f-459e-9b41-fe49955a0e3c"). InnerVolumeSpecName "kube-api-access-hwklp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:18.905068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:18.905033 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwklp\" (UniqueName: \"kubernetes.io/projected/f210a637-436f-459e-9b41-fe49955a0e3c-kube-api-access-hwklp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:19.463249 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.463169 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:03:19.463732 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.463710 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:03:19.464504 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.464485 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:03:19.616812 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.616781 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" Nov 03 23:03:19.617334 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.616809 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5" event={"ID":"f210a637-436f-459e-9b41-fe49955a0e3c","Type":"ContainerDied","Data":"c28ae51b18b39cf86840de79ff1b2c2c399184c467c470373ad9126b01d82df7"} Nov 03 23:03:19.617334 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.616891 4903 scope.go:122] "RemoveContainer" containerID="63438a1f51cfd71ae008bdc5bc25e03115edb624c8d47f632623492278957434" Nov 03 23:03:19.619198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.619180 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" Nov 03 23:03:19.619407 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.619193 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2" event={"ID":"53bd4dbd-7d51-4454-a43b-b8f5680c3857","Type":"ContainerDied","Data":"41951447c0714a15e23fe609d7335fdba0b3c4c118279ed2d20911fa203297dc"} Nov 03 23:03:19.633197 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.632825 4903 scope.go:122] "RemoveContainer" containerID="9b5eb9b6995b80769fab1f89cc42f8dcef1fcb13ba5d0077138d04023432d52d" Nov 03 23:03:19.679283 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.679245 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5"] Nov 03 23:03:19.685720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.685688 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3920cb-754cdc574c-hk2m5"] Nov 03 23:03:19.715267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.715158 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2"] Nov 03 23:03:19.719359 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.719331 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3920cb-57cdc6f756-nl8c2"] Nov 03 23:03:19.735332 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.735281 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53bd4dbd-7d51-4454-a43b-b8f5680c3857" path="/var/lib/kubelet/pods/53bd4dbd-7d51-4454-a43b-b8f5680c3857/volumes" Nov 03 23:03:19.735956 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.735939 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f" path="/var/lib/kubelet/pods/cf2ccb9f-bd02-4ce2-b3ba-3f24c146335f/volumes" Nov 03 23:03:19.736516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:19.736500 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f210a637-436f-459e-9b41-fe49955a0e3c" path="/var/lib/kubelet/pods/f210a637-436f-459e-9b41-fe49955a0e3c/volumes" Nov 03 23:03:39.540173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:39.540139 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9"] Nov 03 23:03:39.545992 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:39.545956 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d4b6bb-khgf9"] Nov 03 23:03:39.737112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:39.737077 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d256e8fe-9234-4879-9126-7c869899ea70" path="/var/lib/kubelet/pods/d256e8fe-9234-4879-9126-7c869899ea70/volumes" Nov 03 23:03:43.705730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.705697 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84"] Nov 03 23:03:43.706666 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706641 4903 cpu_manager.go:414] "containerMap: removing container" podUID="aad1242d-05c2-4478-84fb-d3fe522001a3" containerName="payload-processing" Nov 03 23:03:43.706666 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706665 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad1242d-05c2-4478-84fb-d3fe522001a3" containerName="payload-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706687 4903 cpu_manager.go:414] "containerMap: removing container" podUID="55c3f9f2-b9ee-4605-9854-2e5778c4ad9c" containerName="payload-pre-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706695 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c3f9f2-b9ee-4605-9854-2e5778c4ad9c" containerName="payload-pre-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706710 4903 cpu_manager.go:414] "containerMap: removing container" podUID="53bd4dbd-7d51-4454-a43b-b8f5680c3857" containerName="payload-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706717 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53bd4dbd-7d51-4454-a43b-b8f5680c3857" containerName="payload-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706762 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f210a637-436f-459e-9b41-fe49955a0e3c" containerName="payload-pre-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706769 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f210a637-436f-459e-9b41-fe49955a0e3c" containerName="payload-pre-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706791 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0b50a9c0-0dc4-4b45-9ce4-efc3182d7450" containerName="payload-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706798 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b50a9c0-0dc4-4b45-9ce4-efc3182d7450" containerName="payload-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706819 4903 cpu_manager.go:414] "containerMap: removing container" podUID="17e621b5-3a21-4a9d-b192-dc3c126578a6" containerName="payload-pre-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706826 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e621b5-3a21-4a9d-b192-dc3c126578a6" containerName="payload-pre-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706845 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d256e8fe-9234-4879-9126-7c869899ea70" containerName="revoke-keys" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706853 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d256e8fe-9234-4879-9126-7c869899ea70" containerName="revoke-keys" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706869 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4dfb70f9-51cc-4249-b088-af813d7b54dc" containerName="payload-pre-processing" Nov 03 23:03:43.706889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.706876 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfb70f9-51cc-4249-b088-af813d7b54dc" containerName="payload-pre-processing" Nov 03 23:03:43.707669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.707061 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e621b5-3a21-4a9d-b192-dc3c126578a6" containerName="payload-pre-processing" Nov 03 23:03:43.707669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.707078 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d256e8fe-9234-4879-9126-7c869899ea70" containerName="revoke-keys" Nov 03 23:03:43.707669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.707093 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfb70f9-51cc-4249-b088-af813d7b54dc" containerName="payload-pre-processing" Nov 03 23:03:43.707669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.707104 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad1242d-05c2-4478-84fb-d3fe522001a3" containerName="payload-processing" Nov 03 23:03:43.707669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.707114 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c3f9f2-b9ee-4605-9854-2e5778c4ad9c" containerName="payload-pre-processing" Nov 03 23:03:43.707669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.707124 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="53bd4dbd-7d51-4454-a43b-b8f5680c3857" containerName="payload-processing" Nov 03 23:03:43.707669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.707142 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f210a637-436f-459e-9b41-fe49955a0e3c" containerName="payload-pre-processing" Nov 03 23:03:43.707669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.707152 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b50a9c0-0dc4-4b45-9ce4-efc3182d7450" containerName="payload-processing" Nov 03 23:03:43.711182 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.711126 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.767502 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.767471 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84"] Nov 03 23:03:43.804225 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.804182 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4ad693-openshift-default-dockercfg-nvt7m\"" Nov 03 23:03:43.867093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.867058 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.867286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.867104 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.867286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.867191 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.867286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.867220 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.867286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.867238 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.867497 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.867406 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01285782-5e2a-4be2-aaa3-54910fc5d13f-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.867537 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.867522 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtj7\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-kube-api-access-srtj7\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.867595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.867583 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.867641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.867621 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969357 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969271 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969357 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969340 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969602 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969366 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969602 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969387 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969602 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969403 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969602 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969450 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01285782-5e2a-4be2-aaa3-54910fc5d13f-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969602 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969558 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtj7\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-kube-api-access-srtj7\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969630 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969682 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.969896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969756 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.970017 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.969945 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.970092 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.970070 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.970151 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.970134 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.970337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.970281 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01285782-5e2a-4be2-aaa3-54910fc5d13f-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.971865 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.971846 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.971977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.971958 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.990940 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.990909 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:43.991064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:43.991014 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtj7\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-kube-api-access-srtj7\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:44.055347 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:44.055311 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:44.376409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:44.376377 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84"] Nov 03 23:03:44.376574 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:03:44.376513 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01285782_5e2a_4be2_aaa3_54910fc5d13f.slice/crio-9b3979eb48b7fb5654f950eba1c49768a9ab6780bf91bf6a873fb87f60e4d083 WatchSource:0}: Error finding container 9b3979eb48b7fb5654f950eba1c49768a9ab6780bf91bf6a873fb87f60e4d083: Status 404 returned error can't find the container with id 9b3979eb48b7fb5654f950eba1c49768a9ab6780bf91bf6a873fb87f60e4d083 Nov 03 23:03:44.379266 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:44.379235 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:03:44.379406 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:44.379316 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:03:44.379406 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:44.379350 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:03:44.886216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:44.886174 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" event={"ID":"01285782-5e2a-4be2-aaa3-54910fc5d13f","Type":"ContainerStarted","Data":"30359d989acb4a41a54c9bb5c162f991c81296355eb6b597c789277d378f4be7"} Nov 03 23:03:44.886757 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:44.886242 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" event={"ID":"01285782-5e2a-4be2-aaa3-54910fc5d13f","Type":"ContainerStarted","Data":"9b3979eb48b7fb5654f950eba1c49768a9ab6780bf91bf6a873fb87f60e4d083"} Nov 03 23:03:44.929546 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:44.929471 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" podStartSLOduration=1.929452186 podStartE2EDuration="1.929452186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:44.929082402 +0000 UTC m=+3051.333023377" watchObservedRunningTime="2026-11-03 23:03:44.929452186 +0000 UTC m=+3051.333393161" Nov 03 23:03:45.032557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:45.032508 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4"] Nov 03 23:03:45.036899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:45.036870 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb87ff9e-qrqv4"] Nov 03 23:03:45.065318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:45.061688 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:45.065318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:45.064380 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body= Nov 03 23:03:45.065318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:45.064437 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" Nov 03 23:03:45.777744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:45.777705 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2699ef44-af66-4fa3-8946-94eca4c5ff2e" path="/var/lib/kubelet/pods/2699ef44-af66-4fa3-8946-94eca4c5ff2e/volumes" Nov 03 23:03:46.070597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:46.070499 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body= Nov 03 23:03:46.071179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:46.071134 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" Nov 03 23:03:47.061440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.061401 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body= Nov 03 23:03:47.061831 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.061796 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" Nov 03 23:03:47.609708 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.609672 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4"] Nov 03 23:03:47.613550 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.613532 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.632951 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.632919 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4"] Nov 03 23:03:47.635402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.635375 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.635558 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.635524 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnltk\" (UniqueName: \"kubernetes.io/projected/013bdf85-85c7-428b-a684-f91a637ec72b-kube-api-access-fnltk\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.635706 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.635690 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.637236 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.637213 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:03:47.637359 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.637334 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:03:47.637359 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.637346 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:03:47.637451 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.637364 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:03:47.737377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.737346 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnltk\" (UniqueName: \"kubernetes.io/projected/013bdf85-85c7-428b-a684-f91a637ec72b-kube-api-access-fnltk\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.737560 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.737499 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.737623 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.737561 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.738373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.738343 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.754317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.740644 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.769475 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.768036 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnltk\" (UniqueName: \"kubernetes.io/projected/013bdf85-85c7-428b-a684-f91a637ec72b-kube-api-access-fnltk\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:47.931341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:47.931279 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:48.067402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:48.067016 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body= Nov 03 23:03:48.067402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:48.067074 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" Nov 03 23:03:48.273429 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:03:48.273385 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013bdf85_85c7_428b_a684_f91a637ec72b.slice/crio-dd88e257247848d3f0e4b8d7604fe581cf4ace87c8517a0ea367fd6d9f8389a4 WatchSource:0}: Error finding container dd88e257247848d3f0e4b8d7604fe581cf4ace87c8517a0ea367fd6d9f8389a4: Status 404 returned error can't find the container with id dd88e257247848d3f0e4b8d7604fe581cf4ace87c8517a0ea367fd6d9f8389a4 Nov 03 23:03:48.273638 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:48.273431 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4"] Nov 03 23:03:48.474141 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:03:48.474103 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:03:48.517727 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:48.517684 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:03:48.926240 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:48.926199 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" event={"ID":"013bdf85-85c7-428b-a684-f91a637ec72b","Type":"ContainerStarted","Data":"b22ec8e2c20b5a7b127536297fbc3aaa0f4b121afa8562c12251ccfffc7ff4fe"} Nov 03 23:03:48.926727 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:48.926270 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" event={"ID":"013bdf85-85c7-428b-a684-f91a637ec72b","Type":"ContainerStarted","Data":"dd88e257247848d3f0e4b8d7604fe581cf4ace87c8517a0ea367fd6d9f8389a4"} Nov 03 23:03:48.975368 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:48.975276 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" podStartSLOduration=1.975261908 podStartE2EDuration="1.975261908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:03:48.971672854 +0000 UTC m=+3055.375613826" watchObservedRunningTime="2026-11-03 23:03:48.975261908 +0000 UTC m=+3055.379202882" Nov 03 23:03:49.061369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:49.059400 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body= Nov 03 23:03:49.061369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:49.059484 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" Nov 03 23:03:49.938763 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:49.938728 4903 generic.go:356] "Generic (PLEG): container finished" podID="013bdf85-85c7-428b-a684-f91a637ec72b" containerID="b22ec8e2c20b5a7b127536297fbc3aaa0f4b121afa8562c12251ccfffc7ff4fe" exitCode=0 Nov 03 23:03:49.939177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:49.938778 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" event={"ID":"013bdf85-85c7-428b-a684-f91a637ec72b","Type":"ContainerDied","Data":"b22ec8e2c20b5a7b127536297fbc3aaa0f4b121afa8562c12251ccfffc7ff4fe"} Nov 03 23:03:50.059318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:50.059086 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" start-of-body= Nov 03 23:03:50.059318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:50.059152 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.101:15021/healthz/ready\": dial tcp 10.132.1.101:15021: connect: connection refused" Nov 03 23:03:51.063839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.063579 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:51.065605 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.065581 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:51.066381 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.066358 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:03:51.127536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.127504 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:51.229087 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.228999 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013bdf85-85c7-428b-a684-f91a637ec72b-kube-api-access-fnltk\" (UniqueName: \"kubernetes.io/projected/013bdf85-85c7-428b-a684-f91a637ec72b-kube-api-access-fnltk\") pod \"013bdf85-85c7-428b-a684-f91a637ec72b\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " Nov 03 23:03:51.229087 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.229077 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-xks-ca\") pod \"013bdf85-85c7-428b-a684-f91a637ec72b\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " Nov 03 23:03:51.229394 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.229376 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-openshift-ca\") pod \"013bdf85-85c7-428b-a684-f91a637ec72b\" (UID: \"013bdf85-85c7-428b-a684-f91a637ec72b\") " Nov 03 23:03:51.230121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.230095 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-openshift-ca" pod "013bdf85-85c7-428b-a684-f91a637ec72b" (UID: "013bdf85-85c7-428b-a684-f91a637ec72b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:03:51.232278 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.232251 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-xks-ca" pod "013bdf85-85c7-428b-a684-f91a637ec72b" (UID: "013bdf85-85c7-428b-a684-f91a637ec72b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:03:51.232677 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.232639 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013bdf85-85c7-428b-a684-f91a637ec72b-kube-api-access-fnltk" pod "013bdf85-85c7-428b-a684-f91a637ec72b" (UID: "013bdf85-85c7-428b-a684-f91a637ec72b"). InnerVolumeSpecName "kube-api-access-fnltk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:03:51.330846 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.330814 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:51.330846 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.330843 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnltk\" (UniqueName: \"kubernetes.io/projected/013bdf85-85c7-428b-a684-f91a637ec72b-kube-api-access-fnltk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:51.331048 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.330856 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/013bdf85-85c7-428b-a684-f91a637ec72b-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:03:51.956934 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.956897 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" event={"ID":"013bdf85-85c7-428b-a684-f91a637ec72b","Type":"ContainerDied","Data":"dd88e257247848d3f0e4b8d7604fe581cf4ace87c8517a0ea367fd6d9f8389a4"} Nov 03 23:03:51.956934 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.956928 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4" Nov 03 23:03:51.957195 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:51.956945 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd88e257247848d3f0e4b8d7604fe581cf4ace87c8517a0ea367fd6d9f8389a4" Nov 03 23:03:55.029344 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:55.029307 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4"] Nov 03 23:03:55.039237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:55.039207 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7efbfe-84cz4"] Nov 03 23:03:55.736052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:55.736015 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f97c0b-8429-44fa-8926-fcb9ab390c20" path="/var/lib/kubelet/pods/a6f97c0b-8429-44fa-8926-fcb9ab390c20/volumes" Nov 03 23:03:59.968867 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:59.968832 4903 scope.go:122] "RemoveContainer" containerID="f8ea95357759c3b7f47b20e570c88520abf1405318cad2dacf13191a0c2904f1" Nov 03 23:03:59.983488 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:03:59.983457 4903 scope.go:122] "RemoveContainer" containerID="4ffb30a12bc1fda202efcb58ac0011436f28585f0a278aab55ee9e87a286c9a8" Nov 03 23:04:00.664490 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.664432 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn"] Nov 03 23:04:00.666327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.665789 4903 cpu_manager.go:414] "containerMap: removing container" podUID="013bdf85-85c7-428b-a684-f91a637ec72b" containerName="revoke-keys" Nov 03 23:04:00.666327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.665815 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013bdf85-85c7-428b-a684-f91a637ec72b" containerName="revoke-keys" Nov 03 23:04:00.666327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.666082 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="013bdf85-85c7-428b-a684-f91a637ec72b" containerName="revoke-keys" Nov 03 23:04:00.671974 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.671945 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.688675 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.688643 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn"] Nov 03 23:04:00.690931 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.690900 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:00.691106 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.690938 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:04:00.691182 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.691127 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:04:00.691182 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.691162 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:04:00.766276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.766243 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.766494 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.766413 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.766494 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.766462 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6fjk\" (UniqueName: \"kubernetes.io/projected/b6ee3df3-7300-4c26-be87-a01cba4ce992-kube-api-access-b6fjk\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.867528 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.867485 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.867712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.867551 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fjk\" (UniqueName: \"kubernetes.io/projected/b6ee3df3-7300-4c26-be87-a01cba4ce992-kube-api-access-b6fjk\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.867805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.867731 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.868255 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.868222 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.870281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.870256 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.888978 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.888957 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fjk\" (UniqueName: \"kubernetes.io/projected/b6ee3df3-7300-4c26-be87-a01cba4ce992-kube-api-access-b6fjk\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:00.993138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:00.993061 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:01.301563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:01.301509 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn"] Nov 03 23:04:01.302009 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:04:01.301983 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ee3df3_7300_4c26_be87_a01cba4ce992.slice/crio-db060d5ada9ed97f66b77f242bcf3ecc74a6f5f8fb5fd37bc58656a5ca2efcc6 WatchSource:0}: Error finding container db060d5ada9ed97f66b77f242bcf3ecc74a6f5f8fb5fd37bc58656a5ca2efcc6: Status 404 returned error can't find the container with id db060d5ada9ed97f66b77f242bcf3ecc74a6f5f8fb5fd37bc58656a5ca2efcc6 Nov 03 23:04:02.040438 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:02.040390 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" event={"ID":"b6ee3df3-7300-4c26-be87-a01cba4ce992","Type":"ContainerStarted","Data":"3d9edf6ac45a4b3851ba403eea027f4ab1568100a016860c167f4f7001a72f63"} Nov 03 23:04:02.040938 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:02.040484 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" event={"ID":"b6ee3df3-7300-4c26-be87-a01cba4ce992","Type":"ContainerStarted","Data":"db060d5ada9ed97f66b77f242bcf3ecc74a6f5f8fb5fd37bc58656a5ca2efcc6"} Nov 03 23:04:02.083484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:02.083405 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" podStartSLOduration=2.08338777 podStartE2EDuration="2.08338777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:02.079509031 +0000 UTC m=+3068.483450006" watchObservedRunningTime="2026-11-03 23:04:02.08338777 +0000 UTC m=+3068.487328745" Nov 03 23:04:03.053971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:03.053931 4903 generic.go:356] "Generic (PLEG): container finished" podID="b6ee3df3-7300-4c26-be87-a01cba4ce992" containerID="3d9edf6ac45a4b3851ba403eea027f4ab1568100a016860c167f4f7001a72f63" exitCode=0 Nov 03 23:04:03.054413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:03.054016 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" event={"ID":"b6ee3df3-7300-4c26-be87-a01cba4ce992","Type":"ContainerDied","Data":"3d9edf6ac45a4b3851ba403eea027f4ab1568100a016860c167f4f7001a72f63"} Nov 03 23:04:04.230890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.230858 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:04.318129 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.318034 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-xks-ca\") pod \"b6ee3df3-7300-4c26-be87-a01cba4ce992\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " Nov 03 23:04:04.318348 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.318163 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-openshift-ca\") pod \"b6ee3df3-7300-4c26-be87-a01cba4ce992\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " Nov 03 23:04:04.318348 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.318303 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ee3df3-7300-4c26-be87-a01cba4ce992-kube-api-access-b6fjk\" (UniqueName: \"kubernetes.io/projected/b6ee3df3-7300-4c26-be87-a01cba4ce992-kube-api-access-b6fjk\") pod \"b6ee3df3-7300-4c26-be87-a01cba4ce992\" (UID: \"b6ee3df3-7300-4c26-be87-a01cba4ce992\") " Nov 03 23:04:04.318586 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.318561 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-openshift-ca" pod "b6ee3df3-7300-4c26-be87-a01cba4ce992" (UID: "b6ee3df3-7300-4c26-be87-a01cba4ce992"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:04.318927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.318909 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:04.320144 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.320123 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-xks-ca" pod "b6ee3df3-7300-4c26-be87-a01cba4ce992" (UID: "b6ee3df3-7300-4c26-be87-a01cba4ce992"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:04.320460 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.320440 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ee3df3-7300-4c26-be87-a01cba4ce992-kube-api-access-b6fjk" pod "b6ee3df3-7300-4c26-be87-a01cba4ce992" (UID: "b6ee3df3-7300-4c26-be87-a01cba4ce992"). InnerVolumeSpecName "kube-api-access-b6fjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:04.419911 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.419875 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ee3df3-7300-4c26-be87-a01cba4ce992-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:04.419911 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:04.419906 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6fjk\" (UniqueName: \"kubernetes.io/projected/b6ee3df3-7300-4c26-be87-a01cba4ce992-kube-api-access-b6fjk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:05.071641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:05.071607 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" event={"ID":"b6ee3df3-7300-4c26-be87-a01cba4ce992","Type":"ContainerDied","Data":"db060d5ada9ed97f66b77f242bcf3ecc74a6f5f8fb5fd37bc58656a5ca2efcc6"} Nov 03 23:04:05.071641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:05.071640 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn" Nov 03 23:04:05.071872 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:05.071656 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db060d5ada9ed97f66b77f242bcf3ecc74a6f5f8fb5fd37bc58656a5ca2efcc6" Nov 03 23:04:06.031205 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:06.031165 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh"] Nov 03 23:04:06.041056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:06.041029 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6d61d8-qf8sh"] Nov 03 23:04:07.735363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:07.735328 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7164673-75cb-4497-b91e-8101d8abcbe8" path="/var/lib/kubelet/pods/e7164673-75cb-4497-b91e-8101d8abcbe8/volumes" Nov 03 23:04:08.566456 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:08.566425 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4"] Nov 03 23:04:08.566739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:08.566711 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" containerID="cri-o://775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef" gracePeriod=30 Nov 03 23:04:12.576935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:12.576902 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k"] Nov 03 23:04:12.577554 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:12.577188 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" podUID="7fb02b4d-d081-4caf-82b6-c4f41ae31eb8" containerName="payload-pre-processing" containerID="cri-o://4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5" gracePeriod=30 Nov 03 23:04:12.862489 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:12.862463 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:04:12.944840 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:12.944791 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-kube-api-access-265xn\" (UniqueName: \"kubernetes.io/projected/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-kube-api-access-265xn\") pod \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\" (UID: \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\") " Nov 03 23:04:12.945060 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:12.945020 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-plugins-config-volume\") pod \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\" (UID: \"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8\") " Nov 03 23:04:12.971277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:12.971224 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-plugins-config-volume" pod "7fb02b4d-d081-4caf-82b6-c4f41ae31eb8" (UID: "7fb02b4d-d081-4caf-82b6-c4f41ae31eb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:12.991675 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:12.989904 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-kube-api-access-265xn" pod "7fb02b4d-d081-4caf-82b6-c4f41ae31eb8" (UID: "7fb02b4d-d081-4caf-82b6-c4f41ae31eb8"). InnerVolumeSpecName "kube-api-access-265xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:13.047138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.047096 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-265xn\" (UniqueName: \"kubernetes.io/projected/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-kube-api-access-265xn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.047138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.047131 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.140783 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.140691 4903 generic.go:356] "Generic (PLEG): container finished" podID="7fb02b4d-d081-4caf-82b6-c4f41ae31eb8" containerID="4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5" exitCode=0 Nov 03 23:04:13.140783 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.140743 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" Nov 03 23:04:13.140996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.140778 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" event={"ID":"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8","Type":"ContainerDied","Data":"4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5"} Nov 03 23:04:13.140996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.140835 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k" event={"ID":"7fb02b4d-d081-4caf-82b6-c4f41ae31eb8","Type":"ContainerDied","Data":"583a916aa7835149f1ebaeffbd54e6681fd6d7db1b8ee87206044eb179bf43d9"} Nov 03 23:04:13.140996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.140864 4903 scope.go:122] "RemoveContainer" containerID="4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5" Nov 03 23:04:13.153372 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.153345 4903 scope.go:122] "RemoveContainer" containerID="4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5" Nov 03 23:04:13.153733 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:04:13.153710 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5\": container with ID starting with 4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5 not found: ID does not exist" containerID="4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5" Nov 03 23:04:13.153820 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.153743 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5"} err="failed to get container status \"4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5\": rpc error: code = NotFound desc = could not find container \"4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5\": container with ID starting with 4708b51bf099873243307534f5eaf9cad7b68519c278eedbd01929536a3cffb5 not found: ID does not exist" Nov 03 23:04:13.188635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.188601 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k"] Nov 03 23:04:13.192669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.192642 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4b994-f6b9d5df9-54k5k"] Nov 03 23:04:13.209880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.209850 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn"] Nov 03 23:04:13.213738 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.213714 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4b994-b7jtn"] Nov 03 23:04:13.738102 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.738070 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb02b4d-d081-4caf-82b6-c4f41ae31eb8" path="/var/lib/kubelet/pods/7fb02b4d-d081-4caf-82b6-c4f41ae31eb8/volumes" Nov 03 23:04:13.738627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.738613 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ee3df3-7300-4c26-be87-a01cba4ce992" path="/var/lib/kubelet/pods/b6ee3df3-7300-4c26-be87-a01cba4ce992/volumes" Nov 03 23:04:13.855890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.855858 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r"] Nov 03 23:04:13.856774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.856744 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b6ee3df3-7300-4c26-be87-a01cba4ce992" containerName="revoke-keys" Nov 03 23:04:13.856774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.856765 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ee3df3-7300-4c26-be87-a01cba4ce992" containerName="revoke-keys" Nov 03 23:04:13.856994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.856813 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7fb02b4d-d081-4caf-82b6-c4f41ae31eb8" containerName="payload-pre-processing" Nov 03 23:04:13.856994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.856820 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb02b4d-d081-4caf-82b6-c4f41ae31eb8" containerName="payload-pre-processing" Nov 03 23:04:13.856994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.856937 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ee3df3-7300-4c26-be87-a01cba4ce992" containerName="revoke-keys" Nov 03 23:04:13.856994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.856947 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb02b4d-d081-4caf-82b6-c4f41ae31eb8" containerName="payload-pre-processing" Nov 03 23:04:13.859742 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.859721 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 23:04:13.860578 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.860561 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:13.877390 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.877359 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r"] Nov 03 23:04:13.882537 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.882516 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:04:13.882701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.882560 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:04:13.958147 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.958111 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:13.958354 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.958282 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:04:13.964662 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.964631 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-envoy\") pod \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " Nov 03 23:04:13.964743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.964731 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-data\") pod \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " Nov 03 23:04:13.964788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.964752 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-kube-api-access-4zblt\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-kube-api-access-4zblt\") pod \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " Nov 03 23:04:13.964829 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.964792 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istiod-ca-cert\") pod \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " Nov 03 23:04:13.964865 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.964837 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-token\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-token\") pod \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " Nov 03 23:04:13.964929 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.964914 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-certs\") pod \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " Nov 03 23:04:13.964986 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.964952 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-podinfo\") pod \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " Nov 03 23:04:13.965053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.964991 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-credential-socket\") pod \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " Nov 03 23:04:13.965053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.965025 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-data" pod "04577a03-c0d1-4798-93e7-edc5c3cf1a41" (UID: "04577a03-c0d1-4798-93e7-edc5c3cf1a41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:13.965053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.965041 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-socket\") pod \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\" (UID: \"04577a03-c0d1-4798-93e7-edc5c3cf1a41\") " Nov 03 23:04:13.965355 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.965328 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-credential-socket" pod "04577a03-c0d1-4798-93e7-edc5c3cf1a41" (UID: "04577a03-c0d1-4798-93e7-edc5c3cf1a41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:13.965355 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.965345 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istiod-ca-cert" pod "04577a03-c0d1-4798-93e7-edc5c3cf1a41" (UID: "04577a03-c0d1-4798-93e7-edc5c3cf1a41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:13.965531 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.965429 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdc4k\" (UniqueName: \"kubernetes.io/projected/6cb586d6-540a-4a05-a9b2-71d095ea0180-kube-api-access-cdc4k\") pod \"maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:13.965594 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.965545 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-socket" pod "04577a03-c0d1-4798-93e7-edc5c3cf1a41" (UID: "04577a03-c0d1-4798-93e7-edc5c3cf1a41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:13.965712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.965591 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-certs" pod "04577a03-c0d1-4798-93e7-edc5c3cf1a41" (UID: "04577a03-c0d1-4798-93e7-edc5c3cf1a41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:13.965825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.965802 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:13.965979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.965951 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:13.966094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.966079 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.966178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.966100 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.966178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.966112 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.966178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.966126 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.966325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.966180 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:13.967231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.967206 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-token" pod "04577a03-c0d1-4798-93e7-edc5c3cf1a41" (UID: "04577a03-c0d1-4798-93e7-edc5c3cf1a41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:13.967448 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.967412 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-kube-api-access-4zblt" pod "04577a03-c0d1-4798-93e7-edc5c3cf1a41" (UID: "04577a03-c0d1-4798-93e7-edc5c3cf1a41"). InnerVolumeSpecName "kube-api-access-4zblt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:13.967539 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.967505 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-envoy" pod "04577a03-c0d1-4798-93e7-edc5c3cf1a41" (UID: "04577a03-c0d1-4798-93e7-edc5c3cf1a41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:13.967677 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:13.967659 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-podinfo" pod "04577a03-c0d1-4798-93e7-edc5c3cf1a41" (UID: "04577a03-c0d1-4798-93e7-edc5c3cf1a41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:04:14.067698 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.067662 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:14.067875 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.067759 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:14.067875 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.067825 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdc4k\" (UniqueName: \"kubernetes.io/projected/6cb586d6-540a-4a05-a9b2-71d095ea0180-kube-api-access-cdc4k\") pod \"maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:14.068033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.068014 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:14.068075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.068040 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:14.068075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.068053 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zblt\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-kube-api-access-4zblt\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:14.068075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.068067 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04577a03-c0d1-4798-93e7-edc5c3cf1a41-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:14.068386 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.068364 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:14.070182 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.070166 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:14.089714 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.089643 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdc4k\" (UniqueName: \"kubernetes.io/projected/6cb586d6-540a-4a05-a9b2-71d095ea0180-kube-api-access-cdc4k\") pod \"maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:14.151891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.151855 4903 generic.go:356] "Generic (PLEG): container finished" podID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerID="775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef" exitCode=0 Nov 03 23:04:14.152081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.151909 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" Nov 03 23:04:14.152081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.151938 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" event={"ID":"04577a03-c0d1-4798-93e7-edc5c3cf1a41","Type":"ContainerDied","Data":"775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef"} Nov 03 23:04:14.152081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.151997 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4" event={"ID":"04577a03-c0d1-4798-93e7-edc5c3cf1a41","Type":"ContainerDied","Data":"080ef1f1414056c1f0b9bb8d4f94587874eb8c3d75c4eece4190d3f71f2b12aa"} Nov 03 23:04:14.152081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.152027 4903 scope.go:122] "RemoveContainer" containerID="775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef" Nov 03 23:04:14.164494 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.164474 4903 scope.go:122] "RemoveContainer" containerID="8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63" Nov 03 23:04:14.174491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.174465 4903 scope.go:122] "RemoveContainer" containerID="775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef" Nov 03 23:04:14.174772 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:04:14.174748 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef\": container with ID starting with 775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef not found: ID does not exist" containerID="775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef" Nov 03 23:04:14.174863 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.174778 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef"} err="failed to get container status \"775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef\": rpc error: code = NotFound desc = could not find container \"775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef\": container with ID starting with 775a0e9cb41fdef5bf9b72a1f5b25285364e6088ffe892eab5b34899932a2bef not found: ID does not exist" Nov 03 23:04:14.174863 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.174796 4903 scope.go:122] "RemoveContainer" containerID="8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63" Nov 03 23:04:14.175057 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:04:14.175035 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63\": container with ID starting with 8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63 not found: ID does not exist" containerID="8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63" Nov 03 23:04:14.175105 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.175065 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63"} err="failed to get container status \"8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63\": rpc error: code = NotFound desc = could not find container \"8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63\": container with ID starting with 8bc44508900b6a92fb6546ca8c1365a68cd5fd11b4c128712baa8dec6534ce63 not found: ID does not exist" Nov 03 23:04:14.179702 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.179683 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:14.201351 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.201329 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4"] Nov 03 23:04:14.205631 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.205606 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3920cb-openshift-default-545b848d77-vkcv4"] Nov 03 23:04:14.492042 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:14.492015 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r"] Nov 03 23:04:14.494049 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:04:14.494025 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb586d6_540a_4a05_a9b2_71d095ea0180.slice/crio-00c5248daaf4f77809685c61776ba973bf89c2b0e71490a49da6d3ee5d310f48 WatchSource:0}: Error finding container 00c5248daaf4f77809685c61776ba973bf89c2b0e71490a49da6d3ee5d310f48: Status 404 returned error can't find the container with id 00c5248daaf4f77809685c61776ba973bf89c2b0e71490a49da6d3ee5d310f48 Nov 03 23:04:15.038888 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:15.038774 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd"] Nov 03 23:04:15.042445 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:15.042414 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acab1bd3-79pxd"] Nov 03 23:04:15.169101 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:15.169049 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" event={"ID":"6cb586d6-540a-4a05-a9b2-71d095ea0180","Type":"ContainerStarted","Data":"2b9c6cf78db7b802b10e28808cf527723885ce02122d9b4cc0b6847138fd3e07"} Nov 03 23:04:15.169343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:15.169153 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" event={"ID":"6cb586d6-540a-4a05-a9b2-71d095ea0180","Type":"ContainerStarted","Data":"00c5248daaf4f77809685c61776ba973bf89c2b0e71490a49da6d3ee5d310f48"} Nov 03 23:04:15.212931 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:15.212863 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" podStartSLOduration=2.212844589 podStartE2EDuration="2.212844589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:15.205525748 +0000 UTC m=+3081.609466724" watchObservedRunningTime="2026-11-03 23:04:15.212844589 +0000 UTC m=+3081.616785566" Nov 03 23:04:15.766327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:15.763646 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" path="/var/lib/kubelet/pods/04577a03-c0d1-4798-93e7-edc5c3cf1a41/volumes" Nov 03 23:04:15.766327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:15.764427 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deefbc1a-70c5-4a53-9440-2e6c07d5d6bb" path="/var/lib/kubelet/pods/deefbc1a-70c5-4a53-9440-2e6c07d5d6bb/volumes" Nov 03 23:04:16.183000 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:16.182960 4903 generic.go:356] "Generic (PLEG): container finished" podID="6cb586d6-540a-4a05-a9b2-71d095ea0180" containerID="2b9c6cf78db7b802b10e28808cf527723885ce02122d9b4cc0b6847138fd3e07" exitCode=0 Nov 03 23:04:16.183618 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:16.183055 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" event={"ID":"6cb586d6-540a-4a05-a9b2-71d095ea0180","Type":"ContainerDied","Data":"2b9c6cf78db7b802b10e28808cf527723885ce02122d9b4cc0b6847138fd3e07"} Nov 03 23:04:17.031616 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.031580 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw"] Nov 03 23:04:17.041780 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.041750 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg"] Nov 03 23:04:17.046577 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.046550 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be86daa2-f9wxg"] Nov 03 23:04:17.049903 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.049876 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141ec3d7-rfkvw"] Nov 03 23:04:17.351038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.351004 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:17.412437 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.412400 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-openshift-ca\") pod \"6cb586d6-540a-4a05-a9b2-71d095ea0180\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " Nov 03 23:04:17.412596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.412528 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb586d6-540a-4a05-a9b2-71d095ea0180-kube-api-access-cdc4k\" (UniqueName: \"kubernetes.io/projected/6cb586d6-540a-4a05-a9b2-71d095ea0180-kube-api-access-cdc4k\") pod \"6cb586d6-540a-4a05-a9b2-71d095ea0180\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " Nov 03 23:04:17.412679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.412608 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-xks-ca\") pod \"6cb586d6-540a-4a05-a9b2-71d095ea0180\" (UID: \"6cb586d6-540a-4a05-a9b2-71d095ea0180\") " Nov 03 23:04:17.412760 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.412735 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-openshift-ca" pod "6cb586d6-540a-4a05-a9b2-71d095ea0180" (UID: "6cb586d6-540a-4a05-a9b2-71d095ea0180"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:17.413353 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.413327 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:17.414749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.414722 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb586d6-540a-4a05-a9b2-71d095ea0180-kube-api-access-cdc4k" pod "6cb586d6-540a-4a05-a9b2-71d095ea0180" (UID: "6cb586d6-540a-4a05-a9b2-71d095ea0180"). InnerVolumeSpecName "kube-api-access-cdc4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:17.414870 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.414823 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-xks-ca" pod "6cb586d6-540a-4a05-a9b2-71d095ea0180" (UID: "6cb586d6-540a-4a05-a9b2-71d095ea0180"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:17.514454 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.514422 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdc4k\" (UniqueName: \"kubernetes.io/projected/6cb586d6-540a-4a05-a9b2-71d095ea0180-kube-api-access-cdc4k\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:17.514454 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.514451 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb586d6-540a-4a05-a9b2-71d095ea0180-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:17.736021 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.735988 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d35cbb-4d75-494a-946d-1c5c7caf1b12" path="/var/lib/kubelet/pods/c0d35cbb-4d75-494a-946d-1c5c7caf1b12/volumes" Nov 03 23:04:17.736522 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:17.736505 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5394e7f-9313-4a83-8721-42b8dbc40a34" path="/var/lib/kubelet/pods/e5394e7f-9313-4a83-8721-42b8dbc40a34/volumes" Nov 03 23:04:18.198935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:18.198906 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" Nov 03 23:04:18.199118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:18.198942 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r" event={"ID":"6cb586d6-540a-4a05-a9b2-71d095ea0180","Type":"ContainerDied","Data":"00c5248daaf4f77809685c61776ba973bf89c2b0e71490a49da6d3ee5d310f48"} Nov 03 23:04:18.199118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:18.198994 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00c5248daaf4f77809685c61776ba973bf89c2b0e71490a49da6d3ee5d310f48" Nov 03 23:04:18.384528 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:04:18.384492 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:04:20.621586 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.621552 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j"] Nov 03 23:04:20.622226 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.622210 4903 cpu_manager.go:414] "containerMap: removing container" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" Nov 03 23:04:20.622279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.622227 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" Nov 03 23:04:20.622279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.622246 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6cb586d6-540a-4a05-a9b2-71d095ea0180" containerName="revoke-keys" Nov 03 23:04:20.622279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.622252 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb586d6-540a-4a05-a9b2-71d095ea0180" containerName="revoke-keys" Nov 03 23:04:20.622429 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.622418 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" Nov 03 23:04:20.622467 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.622431 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" Nov 03 23:04:20.622467 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.622441 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb586d6-540a-4a05-a9b2-71d095ea0180" containerName="revoke-keys" Nov 03 23:04:20.627460 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.627442 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.643614 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.643589 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j"] Nov 03 23:04:20.648994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.648968 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:04:20.648994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.648980 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:04:20.715405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.715375 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:20.715405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.715396 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:04:20.761051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.761016 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.761261 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.761059 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87z8c\" (UniqueName: \"kubernetes.io/projected/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-kube-api-access-87z8c\") pod \"maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.761261 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.761216 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.863244 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.863203 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.863499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.863250 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87z8c\" (UniqueName: \"kubernetes.io/projected/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-kube-api-access-87z8c\") pod \"maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.863499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.863367 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.864010 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.863984 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.865905 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.865888 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.885220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.885140 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87z8c\" (UniqueName: \"kubernetes.io/projected/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-kube-api-access-87z8c\") pod \"maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:20.947150 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:20.947113 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:21.253658 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:21.253624 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j"] Nov 03 23:04:21.254939 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:04:21.254914 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fadfd1_495f_46a9_97d9_cf96b7d0c2a6.slice/crio-298bb211fe387483dbd92958680278419d478cfbe2d317bcfe157280c4a0a0ba WatchSource:0}: Error finding container 298bb211fe387483dbd92958680278419d478cfbe2d317bcfe157280c4a0a0ba: Status 404 returned error can't find the container with id 298bb211fe387483dbd92958680278419d478cfbe2d317bcfe157280c4a0a0ba Nov 03 23:04:22.261704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:22.261665 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" event={"ID":"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6","Type":"ContainerStarted","Data":"e9cea010680709d13583251d76561e992474498b83ad4ad34df489c4accf90e0"} Nov 03 23:04:22.262178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:22.261728 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" event={"ID":"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6","Type":"ContainerStarted","Data":"298bb211fe387483dbd92958680278419d478cfbe2d317bcfe157280c4a0a0ba"} Nov 03 23:04:22.311528 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:22.311470 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" podStartSLOduration=2.311456374 podStartE2EDuration="2.311456374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:22.298683233 +0000 UTC m=+3088.702624229" watchObservedRunningTime="2026-11-03 23:04:22.311456374 +0000 UTC m=+3088.715397349" Nov 03 23:04:23.270525 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:23.270489 4903 generic.go:356] "Generic (PLEG): container finished" podID="32fadfd1-495f-46a9-97d9-cf96b7d0c2a6" containerID="e9cea010680709d13583251d76561e992474498b83ad4ad34df489c4accf90e0" exitCode=0 Nov 03 23:04:23.270931 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:23.270572 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" event={"ID":"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6","Type":"ContainerDied","Data":"e9cea010680709d13583251d76561e992474498b83ad4ad34df489c4accf90e0"} Nov 03 23:04:24.432159 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.432130 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:24.619500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.613005 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-xks-ca\") pod \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " Nov 03 23:04:24.619500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.613083 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-kube-api-access-87z8c\" (UniqueName: \"kubernetes.io/projected/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-kube-api-access-87z8c\") pod \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " Nov 03 23:04:24.619500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.613126 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-openshift-ca\") pod \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\" (UID: \"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6\") " Nov 03 23:04:24.619500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.617820 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-openshift-ca" pod "32fadfd1-495f-46a9-97d9-cf96b7d0c2a6" (UID: "32fadfd1-495f-46a9-97d9-cf96b7d0c2a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:24.641938 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.641883 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-xks-ca" pod "32fadfd1-495f-46a9-97d9-cf96b7d0c2a6" (UID: "32fadfd1-495f-46a9-97d9-cf96b7d0c2a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:24.653178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.653122 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-kube-api-access-87z8c" pod "32fadfd1-495f-46a9-97d9-cf96b7d0c2a6" (UID: "32fadfd1-495f-46a9-97d9-cf96b7d0c2a6"). InnerVolumeSpecName "kube-api-access-87z8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:24.715083 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.715046 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:24.715083 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.715080 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87z8c\" (UniqueName: \"kubernetes.io/projected/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-kube-api-access-87z8c\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:24.715315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.715093 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:24.912531 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.912495 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4"] Nov 03 23:04:24.916343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:24.916315 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f9bb0-jm7g4"] Nov 03 23:04:25.289430 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:25.289336 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" event={"ID":"32fadfd1-495f-46a9-97d9-cf96b7d0c2a6","Type":"ContainerDied","Data":"298bb211fe387483dbd92958680278419d478cfbe2d317bcfe157280c4a0a0ba"} Nov 03 23:04:25.289430 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:25.289364 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j" Nov 03 23:04:25.289430 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:25.289400 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298bb211fe387483dbd92958680278419d478cfbe2d317bcfe157280c4a0a0ba" Nov 03 23:04:25.734767 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:25.734734 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013bdf85-85c7-428b-a684-f91a637ec72b" path="/var/lib/kubelet/pods/013bdf85-85c7-428b-a684-f91a637ec72b/volumes" Nov 03 23:04:26.032899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:26.032822 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26"] Nov 03 23:04:26.042606 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:26.042576 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76b008d5-z9n26"] Nov 03 23:04:27.747187 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:27.747153 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003f2579-8b55-4491-abdc-83140f52caba" path="/var/lib/kubelet/pods/003f2579-8b55-4491-abdc-83140f52caba/volumes" Nov 03 23:04:28.031753 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.031672 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt"] Nov 03 23:04:28.043235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.043204 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1207b689-tqskt"] Nov 03 23:04:28.853085 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.853050 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s"] Nov 03 23:04:28.853893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.853858 4903 cpu_manager.go:414] "containerMap: removing container" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" Nov 03 23:04:28.853893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.853881 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04577a03-c0d1-4798-93e7-edc5c3cf1a41" containerName="istio-proxy" Nov 03 23:04:28.854063 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.853902 4903 cpu_manager.go:414] "containerMap: removing container" podUID="32fadfd1-495f-46a9-97d9-cf96b7d0c2a6" containerName="revoke-keys" Nov 03 23:04:28.854063 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.853908 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fadfd1-495f-46a9-97d9-cf96b7d0c2a6" containerName="revoke-keys" Nov 03 23:04:28.854063 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.854052 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fadfd1-495f-46a9-97d9-cf96b7d0c2a6" containerName="revoke-keys" Nov 03 23:04:28.857610 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.857591 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.873797 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.873720 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.873969 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.873826 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e256488-2716-460e-9a4a-e3b9f75014ea-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.873969 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.873928 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.873969 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.873954 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.874175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.873981 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.874175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.874062 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.874175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.874099 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.874408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.874189 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dwq\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-kube-api-access-p7dwq\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.874408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.874247 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.885047 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.885009 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s"] Nov 03 23:04:28.975086 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975047 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dwq\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-kube-api-access-p7dwq\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975101 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975287 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975441 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975387 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e256488-2716-460e-9a4a-e3b9f75014ea-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975509 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975483 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975514 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975652 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975726 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-data\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975845 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975823 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-credential-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975912 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975826 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975912 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975867 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-socket\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.975912 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.975897 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.976129 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.976109 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-certs\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.976198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.976127 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e256488-2716-460e-9a4a-e3b9f75014ea-istiod-ca-cert\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.977775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.977751 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-envoy\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.978288 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.978267 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-podinfo\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.996880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.996854 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-token\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:28.997107 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:28.997086 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dwq\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-kube-api-access-p7dwq\") pod \"e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:29.173075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:29.173046 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:29.498850 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:29.497248 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s"] Nov 03 23:04:29.499359 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:04:29.499328 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e256488_2716_460e_9a4a_e3b9f75014ea.slice/crio-68e99a3eb8c01bac98c6ec3bb8163600cb6625a57d6daf9d233b515259dd51cf WatchSource:0}: Error finding container 68e99a3eb8c01bac98c6ec3bb8163600cb6625a57d6daf9d233b515259dd51cf: Status 404 returned error can't find the container with id 68e99a3eb8c01bac98c6ec3bb8163600cb6625a57d6daf9d233b515259dd51cf Nov 03 23:04:29.502427 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:29.502395 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:04:29.502514 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:29.502460 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:04:29.502514 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:29.502491 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:04:29.736099 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:29.736064 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7004e68d-f98a-4945-bf5b-2d7261c5fb37" path="/var/lib/kubelet/pods/7004e68d-f98a-4945-bf5b-2d7261c5fb37/volumes" Nov 03 23:04:30.331111 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:30.331070 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" event={"ID":"1e256488-2716-460e-9a4a-e3b9f75014ea","Type":"ContainerStarted","Data":"fc696d8760a84e0f40c122009e1e76c1fb90bc415fc11383c6043d5f9d686447"} Nov 03 23:04:30.331567 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:30.331136 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" event={"ID":"1e256488-2716-460e-9a4a-e3b9f75014ea","Type":"ContainerStarted","Data":"68e99a3eb8c01bac98c6ec3bb8163600cb6625a57d6daf9d233b515259dd51cf"} Nov 03 23:04:30.403873 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:30.403818 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" podStartSLOduration=2.403802892 podStartE2EDuration="2.403802892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:30.392972295 +0000 UTC m=+3096.796913283" watchObservedRunningTime="2026-11-03 23:04:30.403802892 +0000 UTC m=+3096.807743867" Nov 03 23:04:31.173384 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:31.173345 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:31.175278 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:31.175246 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 03 23:04:31.175464 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:31.175311 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 03 23:04:32.173636 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.173601 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 03 23:04:32.174093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.173664 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 03 23:04:32.823443 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.823396 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p"] Nov 03 23:04:32.829039 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.829017 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:32.876399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.862177 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:04:32.876399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.862286 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:04:32.895396 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.890121 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:04:32.895396 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.890195 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:32.895396 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.890465 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p"] Nov 03 23:04:32.965317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.965127 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459552e8-0778-4a23-a363-52182defd2da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:32.965526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.965448 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459552e8-0778-4a23-a363-52182defd2da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:32.965747 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:32.965725 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks42f\" (UniqueName: \"kubernetes.io/projected/459552e8-0778-4a23-a363-52182defd2da-kube-api-access-ks42f\") pod \"maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:33.067241 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.067199 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks42f\" (UniqueName: \"kubernetes.io/projected/459552e8-0778-4a23-a363-52182defd2da-kube-api-access-ks42f\") pod \"maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:33.067483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.067275 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459552e8-0778-4a23-a363-52182defd2da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:33.067483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.067354 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459552e8-0778-4a23-a363-52182defd2da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:33.068186 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.068150 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459552e8-0778-4a23-a363-52182defd2da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:33.070065 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.070038 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459552e8-0778-4a23-a363-52182defd2da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:33.088745 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.088685 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks42f\" (UniqueName: \"kubernetes.io/projected/459552e8-0778-4a23-a363-52182defd2da-kube-api-access-ks42f\") pod \"maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:33.173705 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.173670 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 03 23:04:33.174182 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.173742 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 03 23:04:33.175861 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.175834 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:33.517035 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:33.517006 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p"] Nov 03 23:04:33.518259 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:04:33.518233 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459552e8_0778_4a23_a363_52182defd2da.slice/crio-eb74ad60bf7369442190e7171de2cee63ca304a3e4cab9ab25cf602ae7ba0293 WatchSource:0}: Error finding container eb74ad60bf7369442190e7171de2cee63ca304a3e4cab9ab25cf602ae7ba0293: Status 404 returned error can't find the container with id eb74ad60bf7369442190e7171de2cee63ca304a3e4cab9ab25cf602ae7ba0293 Nov 03 23:04:34.174345 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:34.174279 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 03 23:04:34.174749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:34.174369 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 03 23:04:34.391700 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:34.391664 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" event={"ID":"459552e8-0778-4a23-a363-52182defd2da","Type":"ContainerStarted","Data":"e4131cc25c8a90f8a348f9285555f895471557f54bbea51434b586b6479d42f6"} Nov 03 23:04:34.391889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:34.391724 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" event={"ID":"459552e8-0778-4a23-a363-52182defd2da","Type":"ContainerStarted","Data":"eb74ad60bf7369442190e7171de2cee63ca304a3e4cab9ab25cf602ae7ba0293"} Nov 03 23:04:34.428557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:34.428354 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" podStartSLOduration=2.428335396 podStartE2EDuration="2.428335396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:34.427229869 +0000 UTC m=+3100.831170845" watchObservedRunningTime="2026-11-03 23:04:34.428335396 +0000 UTC m=+3100.832276374" Nov 03 23:04:35.174175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:35.174142 4903 patch_prober.go:29] interesting pod/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body= Nov 03 23:04:35.174392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:35.174194 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" Nov 03 23:04:35.400779 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:35.400735 4903 generic.go:356] "Generic (PLEG): container finished" podID="459552e8-0778-4a23-a363-52182defd2da" containerID="e4131cc25c8a90f8a348f9285555f895471557f54bbea51434b586b6479d42f6" exitCode=0 Nov 03 23:04:35.400979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:35.400824 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" event={"ID":"459552e8-0778-4a23-a363-52182defd2da","Type":"ContainerDied","Data":"e4131cc25c8a90f8a348f9285555f895471557f54bbea51434b586b6479d42f6"} Nov 03 23:04:36.179412 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.179334 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:36.179844 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.179777 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:36.180526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.180501 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:04:36.591128 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.591091 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:36.730173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.730079 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459552e8-0778-4a23-a363-52182defd2da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459552e8-0778-4a23-a363-52182defd2da-maas-api-openshift-ca\") pod \"459552e8-0778-4a23-a363-52182defd2da\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " Nov 03 23:04:36.730173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.730145 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459552e8-0778-4a23-a363-52182defd2da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459552e8-0778-4a23-a363-52182defd2da-maas-api-xks-ca\") pod \"459552e8-0778-4a23-a363-52182defd2da\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " Nov 03 23:04:36.730426 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.730272 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459552e8-0778-4a23-a363-52182defd2da-kube-api-access-ks42f\" (UniqueName: \"kubernetes.io/projected/459552e8-0778-4a23-a363-52182defd2da-kube-api-access-ks42f\") pod \"459552e8-0778-4a23-a363-52182defd2da\" (UID: \"459552e8-0778-4a23-a363-52182defd2da\") " Nov 03 23:04:36.730534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.730508 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459552e8-0778-4a23-a363-52182defd2da-maas-api-openshift-ca" pod "459552e8-0778-4a23-a363-52182defd2da" (UID: "459552e8-0778-4a23-a363-52182defd2da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:36.730819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.730795 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459552e8-0778-4a23-a363-52182defd2da-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:36.732469 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.732437 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459552e8-0778-4a23-a363-52182defd2da-maas-api-xks-ca" pod "459552e8-0778-4a23-a363-52182defd2da" (UID: "459552e8-0778-4a23-a363-52182defd2da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:36.732682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.732660 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459552e8-0778-4a23-a363-52182defd2da-kube-api-access-ks42f" pod "459552e8-0778-4a23-a363-52182defd2da" (UID: "459552e8-0778-4a23-a363-52182defd2da"). InnerVolumeSpecName "kube-api-access-ks42f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:36.832397 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.832358 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks42f\" (UniqueName: \"kubernetes.io/projected/459552e8-0778-4a23-a363-52182defd2da-kube-api-access-ks42f\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:36.832397 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:36.832386 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459552e8-0778-4a23-a363-52182defd2da-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:37.417649 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:37.417619 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" Nov 03 23:04:37.418047 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:37.417652 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p" event={"ID":"459552e8-0778-4a23-a363-52182defd2da","Type":"ContainerDied","Data":"eb74ad60bf7369442190e7171de2cee63ca304a3e4cab9ab25cf602ae7ba0293"} Nov 03 23:04:37.418047 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:37.417704 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb74ad60bf7369442190e7171de2cee63ca304a3e4cab9ab25cf602ae7ba0293" Nov 03 23:04:37.589946 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:37.589907 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq"] Nov 03 23:04:37.594629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:37.594597 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf"] Nov 03 23:04:43.276684 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:43.276645 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2"] Nov 03 23:04:48.277836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:48.277770 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" podUID="a7ec5663-6735-4dab-b18d-a3f3b722c2a2" containerName="payload-pre-processing" containerID="cri-o://eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd" gracePeriod=25 Nov 03 23:04:48.384336 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:04:48.384264 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:04:51.307657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.307623 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f"] Nov 03 23:04:51.308175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.307831 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" podUID="b2341ba1-d0ba-460a-ab1e-58a6fd17e217" containerName="payload-processing" containerID="cri-o://63fd0b86e9622ac137e14ec25c3fbef3c4b88c31c37e3017b9aed04db32404aa" gracePeriod=30 Nov 03 23:04:51.329563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.329526 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8"] Nov 03 23:04:51.329976 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.329944 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" podUID="13fdf073-9c1e-4695-8ce6-af0287391ef3" containerName="payload-pre-processing" containerID="cri-o://afe482f5b49c809ea69d9b48fbf412ef2f3e5382bcc6aede610399770197da0f" gracePeriod=30 Nov 03 23:04:51.564480 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.563562 4903 generic.go:356] "Generic (PLEG): container finished" podID="13fdf073-9c1e-4695-8ce6-af0287391ef3" containerID="afe482f5b49c809ea69d9b48fbf412ef2f3e5382bcc6aede610399770197da0f" exitCode=0 Nov 03 23:04:51.564480 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.563867 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" event={"ID":"13fdf073-9c1e-4695-8ce6-af0287391ef3","Type":"ContainerDied","Data":"afe482f5b49c809ea69d9b48fbf412ef2f3e5382bcc6aede610399770197da0f"} Nov 03 23:04:51.567093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.566684 4903 generic.go:356] "Generic (PLEG): container finished" podID="b2341ba1-d0ba-460a-ab1e-58a6fd17e217" containerID="63fd0b86e9622ac137e14ec25c3fbef3c4b88c31c37e3017b9aed04db32404aa" exitCode=0 Nov 03 23:04:51.567093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.566811 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" event={"ID":"b2341ba1-d0ba-460a-ab1e-58a6fd17e217","Type":"ContainerDied","Data":"63fd0b86e9622ac137e14ec25c3fbef3c4b88c31c37e3017b9aed04db32404aa"} Nov 03 23:04:51.598338 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.594528 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq"] Nov 03 23:04:51.598338 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.596490 4903 cpu_manager.go:414] "containerMap: removing container" podUID="459552e8-0778-4a23-a363-52182defd2da" containerName="revoke-keys" Nov 03 23:04:51.598338 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.596510 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459552e8-0778-4a23-a363-52182defd2da" containerName="revoke-keys" Nov 03 23:04:51.598338 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.596671 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="459552e8-0778-4a23-a363-52182defd2da" containerName="revoke-keys" Nov 03 23:04:51.602038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.602015 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.610456 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.610428 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr"] Nov 03 23:04:51.615288 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.615264 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:04:51.615541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.615360 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:04:51.615541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.615366 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:51.615541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.615483 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:04:51.615830 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.615808 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.616504 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.616483 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq"] Nov 03 23:04:51.650720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.645828 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6edce4-g5srq\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.650720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.646036 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr"] Nov 03 23:04:51.650720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.646087 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6edce4-g5srq\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.650720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.646134 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qp8\" (UniqueName: \"kubernetes.io/projected/0bf36ae2-a437-4a1e-adc5-b554beb18795-kube-api-access-l8qp8\") pod \"maas-api-revoke-sub-bf6edce4-g5srq\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.661983 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.661959 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:04:51.702018 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.701992 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:04:51.747638 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.747598 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-kube-api-access-hhnmx\" (UniqueName: \"kubernetes.io/projected/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-kube-api-access-hhnmx\") pod \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\" (UID: \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\") " Nov 03 23:04:51.747883 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.747710 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fdf073-9c1e-4695-8ce6-af0287391ef3-kube-api-access-nnfxd\" (UniqueName: \"kubernetes.io/projected/13fdf073-9c1e-4695-8ce6-af0287391ef3-kube-api-access-nnfxd\") pod \"13fdf073-9c1e-4695-8ce6-af0287391ef3\" (UID: \"13fdf073-9c1e-4695-8ce6-af0287391ef3\") " Nov 03 23:04:51.747883 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.747764 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fdf073-9c1e-4695-8ce6-af0287391ef3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fdf073-9c1e-4695-8ce6-af0287391ef3-plugins-config-volume\") pod \"13fdf073-9c1e-4695-8ce6-af0287391ef3\" (UID: \"13fdf073-9c1e-4695-8ce6-af0287391ef3\") " Nov 03 23:04:51.747883 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.747869 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-plugins-config-volume\") pod \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\" (UID: \"b2341ba1-d0ba-460a-ab1e-58a6fd17e217\") " Nov 03 23:04:51.748181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.748148 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a23ddf01-7p7jr\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.748288 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.748220 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6edce4-g5srq\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.748288 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.748258 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qp8\" (UniqueName: \"kubernetes.io/projected/0bf36ae2-a437-4a1e-adc5-b554beb18795-kube-api-access-l8qp8\") pod \"maas-api-revoke-sub-bf6edce4-g5srq\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.748508 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.748347 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a23ddf01-7p7jr\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.748508 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.748380 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lvj5\" (UniqueName: \"kubernetes.io/projected/2709af96-abe6-4467-b9ad-e32c7dd469aa-kube-api-access-7lvj5\") pod \"maas-api-revoke-sub-a23ddf01-7p7jr\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.748508 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.748493 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6edce4-g5srq\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.749514 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.749142 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-plugins-config-volume" pod "b2341ba1-d0ba-460a-ab1e-58a6fd17e217" (UID: "b2341ba1-d0ba-460a-ab1e-58a6fd17e217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:51.749514 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.749262 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf6edce4-g5srq\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.750615 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.750577 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-kube-api-access-hhnmx" pod "b2341ba1-d0ba-460a-ab1e-58a6fd17e217" (UID: "b2341ba1-d0ba-460a-ab1e-58a6fd17e217"). InnerVolumeSpecName "kube-api-access-hhnmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:51.751124 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.751101 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fdf073-9c1e-4695-8ce6-af0287391ef3-plugins-config-volume" pod "13fdf073-9c1e-4695-8ce6-af0287391ef3" (UID: "13fdf073-9c1e-4695-8ce6-af0287391ef3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:51.751649 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.751617 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fdf073-9c1e-4695-8ce6-af0287391ef3-kube-api-access-nnfxd" pod "13fdf073-9c1e-4695-8ce6-af0287391ef3" (UID: "13fdf073-9c1e-4695-8ce6-af0287391ef3"). InnerVolumeSpecName "kube-api-access-nnfxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:51.756818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.754575 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf6edce4-g5srq\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.776934 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.776902 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qp8\" (UniqueName: \"kubernetes.io/projected/0bf36ae2-a437-4a1e-adc5-b554beb18795-kube-api-access-l8qp8\") pod \"maas-api-revoke-sub-bf6edce4-g5srq\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.850545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.850440 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a23ddf01-7p7jr\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.850726 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.850628 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a23ddf01-7p7jr\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.850726 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.850664 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvj5\" (UniqueName: \"kubernetes.io/projected/2709af96-abe6-4467-b9ad-e32c7dd469aa-kube-api-access-7lvj5\") pod \"maas-api-revoke-sub-a23ddf01-7p7jr\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.850889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.850868 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:51.850971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.850896 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhnmx\" (UniqueName: \"kubernetes.io/projected/b2341ba1-d0ba-460a-ab1e-58a6fd17e217-kube-api-access-hhnmx\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:51.850971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.850911 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnfxd\" (UniqueName: \"kubernetes.io/projected/13fdf073-9c1e-4695-8ce6-af0287391ef3-kube-api-access-nnfxd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:51.850971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.850928 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13fdf073-9c1e-4695-8ce6-af0287391ef3-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:51.852910 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.852876 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a23ddf01-7p7jr\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.854486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.854461 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a23ddf01-7p7jr\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.881258 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.881224 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r"] Nov 03 23:04:51.885011 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.884978 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvj5\" (UniqueName: \"kubernetes.io/projected/2709af96-abe6-4467-b9ad-e32c7dd469aa-kube-api-access-7lvj5\") pod \"maas-api-revoke-sub-a23ddf01-7p7jr\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:51.885690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.885651 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f48f46-tfk8r"] Nov 03 23:04:51.990856 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.990827 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:51.999414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:51.999385 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:52.049563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.049524 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:04:52.523575 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.523543 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq"] Nov 03 23:04:52.523575 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:04:52.523560 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf36ae2_a437_4a1e_adc5_b554beb18795.slice/crio-cff73936f57bb59480e87537d6cbc8e93a9a2c2b71db95643b854aa88d3d3886 WatchSource:0}: Error finding container cff73936f57bb59480e87537d6cbc8e93a9a2c2b71db95643b854aa88d3d3886: Status 404 returned error can't find the container with id cff73936f57bb59480e87537d6cbc8e93a9a2c2b71db95643b854aa88d3d3886 Nov 03 23:04:52.547529 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:04:52.547481 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2709af96_abe6_4467_b9ad_e32c7dd469aa.slice/crio-c43c16f135fd92b559ece376d74f6548493d9d27b51ae7201cb9448d15aad974 WatchSource:0}: Error finding container c43c16f135fd92b559ece376d74f6548493d9d27b51ae7201cb9448d15aad974: Status 404 returned error can't find the container with id c43c16f135fd92b559ece376d74f6548493d9d27b51ae7201cb9448d15aad974 Nov 03 23:04:52.547529 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.547510 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr"] Nov 03 23:04:52.578746 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.578725 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" Nov 03 23:04:52.578891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.578739 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8" event={"ID":"13fdf073-9c1e-4695-8ce6-af0287391ef3","Type":"ContainerDied","Data":"9954c8debf809286b2fbd800d92430f8a8b601a5ce71f52e86eea2bc6a6393d3"} Nov 03 23:04:52.578891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.578807 4903 scope.go:122] "RemoveContainer" containerID="afe482f5b49c809ea69d9b48fbf412ef2f3e5382bcc6aede610399770197da0f" Nov 03 23:04:52.584230 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.584173 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" event={"ID":"2709af96-abe6-4467-b9ad-e32c7dd469aa","Type":"ContainerStarted","Data":"c43c16f135fd92b559ece376d74f6548493d9d27b51ae7201cb9448d15aad974"} Nov 03 23:04:52.585958 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.585940 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" Nov 03 23:04:52.586053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.585952 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f" event={"ID":"b2341ba1-d0ba-460a-ab1e-58a6fd17e217","Type":"ContainerDied","Data":"738437fe8637fb183ba28f22a00e378fa007403d4521d4e888a59b4825890008"} Nov 03 23:04:52.587514 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.587460 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" event={"ID":"0bf36ae2-a437-4a1e-adc5-b554beb18795","Type":"ContainerStarted","Data":"cff73936f57bb59480e87537d6cbc8e93a9a2c2b71db95643b854aa88d3d3886"} Nov 03 23:04:52.596990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.596953 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j"] Nov 03 23:04:52.603044 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.603013 4903 scope.go:122] "RemoveContainer" containerID="63fd0b86e9622ac137e14ec25c3fbef3c4b88c31c37e3017b9aed04db32404aa" Nov 03 23:04:52.603801 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.603777 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-162d7b-d568j"] Nov 03 23:04:52.612342 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.612308 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" containerID="cri-o://b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486" gracePeriod=45 Nov 03 23:04:52.615281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.615177 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" containerID="cri-o://4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e" gracePeriod=45 Nov 03 23:04:52.677237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.677134 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f"] Nov 03 23:04:52.681408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.681372 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f48f46-7f5988976-2zk5f"] Nov 03 23:04:52.693288 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.693262 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8"] Nov 03 23:04:52.724617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.724588 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f48f46-5bbdc5f47f-9x7w8"] Nov 03 23:04:52.957478 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:52.957454 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 23:04:53.084196 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.084101 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f3c564-4549-49be-aba0-e474d0ad07aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3c564-4549-49be-aba0-e474d0ad07aa-tls-certs\") pod \"d8f3c564-4549-49be-aba0-e474d0ad07aa\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " Nov 03 23:04:53.084196 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.084178 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-tmp-dir\") pod \"d8f3c564-4549-49be-aba0-e474d0ad07aa\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " Nov 03 23:04:53.084466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.084252 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-model-cache\") pod \"d8f3c564-4549-49be-aba0-e474d0ad07aa\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " Nov 03 23:04:53.084466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.084372 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f3c564-4549-49be-aba0-e474d0ad07aa-kube-api-access-4ts8v\" (UniqueName: \"kubernetes.io/projected/d8f3c564-4549-49be-aba0-e474d0ad07aa-kube-api-access-4ts8v\") pod \"d8f3c564-4549-49be-aba0-e474d0ad07aa\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " Nov 03 23:04:53.084597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.084489 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-tmp-dir" pod "d8f3c564-4549-49be-aba0-e474d0ad07aa" (UID: "d8f3c564-4549-49be-aba0-e474d0ad07aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:53.084597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.084534 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-dshm\") pod \"d8f3c564-4549-49be-aba0-e474d0ad07aa\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " Nov 03 23:04:53.084717 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.084627 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-home\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-home\") pod \"d8f3c564-4549-49be-aba0-e474d0ad07aa\" (UID: \"d8f3c564-4549-49be-aba0-e474d0ad07aa\") " Nov 03 23:04:53.084780 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.084731 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-model-cache" pod "d8f3c564-4549-49be-aba0-e474d0ad07aa" (UID: "d8f3c564-4549-49be-aba0-e474d0ad07aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:53.084953 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.084936 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-home" pod "d8f3c564-4549-49be-aba0-e474d0ad07aa" (UID: "d8f3c564-4549-49be-aba0-e474d0ad07aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:53.085369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.085285 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.085369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.085329 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.085369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.085346 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.090697 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.090670 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f3c564-4549-49be-aba0-e474d0ad07aa-tls-certs" pod "d8f3c564-4549-49be-aba0-e474d0ad07aa" (UID: "d8f3c564-4549-49be-aba0-e474d0ad07aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:53.090835 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.090731 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f3c564-4549-49be-aba0-e474d0ad07aa-kube-api-access-4ts8v" pod "d8f3c564-4549-49be-aba0-e474d0ad07aa" (UID: "d8f3c564-4549-49be-aba0-e474d0ad07aa"). InnerVolumeSpecName "kube-api-access-4ts8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:53.093020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.092993 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-dshm" pod "d8f3c564-4549-49be-aba0-e474d0ad07aa" (UID: "d8f3c564-4549-49be-aba0-e474d0ad07aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:53.123321 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.123282 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 23:04:53.186658 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.186626 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-model-cache\") pod \"2778075f-b2e9-4e82-b8ac-7489c9a55918\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " Nov 03 23:04:53.186825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.186698 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2778075f-b2e9-4e82-b8ac-7489c9a55918-tls-certs\" (UniqueName: \"kubernetes.io/secret/2778075f-b2e9-4e82-b8ac-7489c9a55918-tls-certs\") pod \"2778075f-b2e9-4e82-b8ac-7489c9a55918\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " Nov 03 23:04:53.186895 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.186844 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2778075f-b2e9-4e82-b8ac-7489c9a55918-kube-api-access-cngx5\" (UniqueName: \"kubernetes.io/projected/2778075f-b2e9-4e82-b8ac-7489c9a55918-kube-api-access-cngx5\") pod \"2778075f-b2e9-4e82-b8ac-7489c9a55918\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " Nov 03 23:04:53.186895 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.186876 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-tmp-dir\") pod \"2778075f-b2e9-4e82-b8ac-7489c9a55918\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " Nov 03 23:04:53.187014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.186919 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-model-cache" pod "2778075f-b2e9-4e82-b8ac-7489c9a55918" (UID: "2778075f-b2e9-4e82-b8ac-7489c9a55918"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:53.187014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.186932 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-dshm\") pod \"2778075f-b2e9-4e82-b8ac-7489c9a55918\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " Nov 03 23:04:53.187134 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.187055 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-home\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-home\") pod \"2778075f-b2e9-4e82-b8ac-7489c9a55918\" (UID: \"2778075f-b2e9-4e82-b8ac-7489c9a55918\") " Nov 03 23:04:53.187734 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.187706 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f3c564-4549-49be-aba0-e474d0ad07aa-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.187734 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.187729 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.187973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.187745 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f3c564-4549-49be-aba0-e474d0ad07aa-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.187973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.187759 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ts8v\" (UniqueName: \"kubernetes.io/projected/d8f3c564-4549-49be-aba0-e474d0ad07aa-kube-api-access-4ts8v\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.188156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.187972 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-home" pod "2778075f-b2e9-4e82-b8ac-7489c9a55918" (UID: "2778075f-b2e9-4e82-b8ac-7489c9a55918"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:53.188156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.188135 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-tmp-dir" pod "2778075f-b2e9-4e82-b8ac-7489c9a55918" (UID: "2778075f-b2e9-4e82-b8ac-7489c9a55918"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:53.189094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.189068 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-dshm" pod "2778075f-b2e9-4e82-b8ac-7489c9a55918" (UID: "2778075f-b2e9-4e82-b8ac-7489c9a55918"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:04:53.189263 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.189247 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2778075f-b2e9-4e82-b8ac-7489c9a55918-tls-certs" pod "2778075f-b2e9-4e82-b8ac-7489c9a55918" (UID: "2778075f-b2e9-4e82-b8ac-7489c9a55918"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:53.189528 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.189511 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2778075f-b2e9-4e82-b8ac-7489c9a55918-kube-api-access-cngx5" pod "2778075f-b2e9-4e82-b8ac-7489c9a55918" (UID: "2778075f-b2e9-4e82-b8ac-7489c9a55918"). InnerVolumeSpecName "kube-api-access-cngx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:53.290626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.290484 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cngx5\" (UniqueName: \"kubernetes.io/projected/2778075f-b2e9-4e82-b8ac-7489c9a55918-kube-api-access-cngx5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.290626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.290523 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.290626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.290542 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.290626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.290556 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2778075f-b2e9-4e82-b8ac-7489c9a55918-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.290626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.290570 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2778075f-b2e9-4e82-b8ac-7489c9a55918-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:53.606141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.605850 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" event={"ID":"0bf36ae2-a437-4a1e-adc5-b554beb18795","Type":"ContainerStarted","Data":"3372923062b5db0a8c2a55123f5d5dcf9bce6e202b40a657a04a6833e5b5f680"} Nov 03 23:04:53.607862 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.607837 4903 generic.go:356] "Generic (PLEG): container finished" podID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerID="4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e" exitCode=0 Nov 03 23:04:53.608073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.607888 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" Nov 03 23:04:53.608635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.607908 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" event={"ID":"2778075f-b2e9-4e82-b8ac-7489c9a55918","Type":"ContainerDied","Data":"4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e"} Nov 03 23:04:53.608761 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.608653 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf" event={"ID":"2778075f-b2e9-4e82-b8ac-7489c9a55918","Type":"ContainerDied","Data":"3d08d0c35e85e2351350dd7ab889ab3e0e36ea8177bdac995942031f3cbb8cd9"} Nov 03 23:04:53.608761 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.608679 4903 scope.go:122] "RemoveContainer" containerID="4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e" Nov 03 23:04:53.617574 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.617551 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" event={"ID":"2709af96-abe6-4467-b9ad-e32c7dd469aa","Type":"ContainerStarted","Data":"84f67e14d4cba0f87290d252265eb26e3ce6c85c58da1b54e17b5a10df595b49"} Nov 03 23:04:53.619171 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.619150 4903 generic.go:356] "Generic (PLEG): container finished" podID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerID="b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486" exitCode=0 Nov 03 23:04:53.619283 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.619193 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" Nov 03 23:04:53.619283 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.619232 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" event={"ID":"d8f3c564-4549-49be-aba0-e474d0ad07aa","Type":"ContainerDied","Data":"b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486"} Nov 03 23:04:53.619283 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.619269 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq" event={"ID":"d8f3c564-4549-49be-aba0-e474d0ad07aa","Type":"ContainerDied","Data":"9af170a381b40aa12d45cb0b26b9716f6e134e263043ab98111d9c38e598400b"} Nov 03 23:04:53.646627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.646568 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" podStartSLOduration=2.646549844 podStartE2EDuration="2.646549844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:04:53.64169749 +0000 UTC m=+3120.045638479" watchObservedRunningTime="2026-11-03 23:04:53.646549844 +0000 UTC m=+3120.050490819" Nov 03 23:04:53.721443 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.721418 4903 scope.go:122] "RemoveContainer" containerID="4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e" Nov 03 23:04:53.721829 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:04:53.721798 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e\": container with ID starting with 4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e not found: ID does not exist" containerID="4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e" Nov 03 23:04:53.721941 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.721834 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e"} err="failed to get container status \"4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e\": rpc error: code = NotFound desc = could not find container \"4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e\": container with ID starting with 4d5f94426ca6b2c30346b0609125406cf9ee1e993b5fb8fe63b33e9150f5ea2e not found: ID does not exist" Nov 03 23:04:53.721941 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.721852 4903 scope.go:122] "RemoveContainer" containerID="b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486" Nov 03 23:04:53.732640 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.732618 4903 scope.go:122] "RemoveContainer" containerID="b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486" Nov 03 23:04:53.732933 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:04:53.732904 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486\": container with ID starting with b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486 not found: ID does not exist" containerID="b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486" Nov 03 23:04:53.733035 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.732942 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486"} err="failed to get container status \"b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486\": rpc error: code = NotFound desc = could not find container \"b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486\": container with ID starting with b533f22d8a836ffe5afa555212dacc455a6b0e7fa441f7681e8c7dc0515d4486 not found: ID does not exist" Nov 03 23:04:53.737001 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.736976 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13fdf073-9c1e-4695-8ce6-af0287391ef3" path="/var/lib/kubelet/pods/13fdf073-9c1e-4695-8ce6-af0287391ef3/volumes" Nov 03 23:04:53.737625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.737606 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fadfd1-495f-46a9-97d9-cf96b7d0c2a6" path="/var/lib/kubelet/pods/32fadfd1-495f-46a9-97d9-cf96b7d0c2a6/volumes" Nov 03 23:04:53.738179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.738165 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb586d6-540a-4a05-a9b2-71d095ea0180" path="/var/lib/kubelet/pods/6cb586d6-540a-4a05-a9b2-71d095ea0180/volumes" Nov 03 23:04:53.739056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.739039 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2341ba1-d0ba-460a-ab1e-58a6fd17e217" path="/var/lib/kubelet/pods/b2341ba1-d0ba-460a-ab1e-58a6fd17e217/volumes" Nov 03 23:04:53.767314 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.760531 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf"] Nov 03 23:04:53.771872 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.771829 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-553753/premium-w2-553753-kserve-59b6d6799b-z57gf"] Nov 03 23:04:53.796250 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.796211 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq"] Nov 03 23:04:53.796478 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:53.796263 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-553753/distinct-w2-553753-kserve-797c58bb75-tf6rq"] Nov 03 23:04:54.185237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.185193 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" podUID="a7ec5663-6735-4dab-b18d-a3f3b722c2a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:04:54.628900 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.628818 4903 generic.go:356] "Generic (PLEG): container finished" podID="0bf36ae2-a437-4a1e-adc5-b554beb18795" containerID="3372923062b5db0a8c2a55123f5d5dcf9bce6e202b40a657a04a6833e5b5f680" exitCode=0 Nov 03 23:04:54.629325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.628903 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" event={"ID":"0bf36ae2-a437-4a1e-adc5-b554beb18795","Type":"ContainerDied","Data":"3372923062b5db0a8c2a55123f5d5dcf9bce6e202b40a657a04a6833e5b5f680"} Nov 03 23:04:54.631808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.631785 4903 generic.go:356] "Generic (PLEG): container finished" podID="2709af96-abe6-4467-b9ad-e32c7dd469aa" containerID="84f67e14d4cba0f87290d252265eb26e3ce6c85c58da1b54e17b5a10df595b49" exitCode=0 Nov 03 23:04:54.631928 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.631828 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" event={"ID":"2709af96-abe6-4467-b9ad-e32c7dd469aa","Type":"ContainerDied","Data":"84f67e14d4cba0f87290d252265eb26e3ce6c85c58da1b54e17b5a10df595b49"} Nov 03 23:04:54.797218 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.797192 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:54.930444 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.930400 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2709af96-abe6-4467-b9ad-e32c7dd469aa-kube-api-access-7lvj5\" (UniqueName: \"kubernetes.io/projected/2709af96-abe6-4467-b9ad-e32c7dd469aa-kube-api-access-7lvj5\") pod \"2709af96-abe6-4467-b9ad-e32c7dd469aa\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " Nov 03 23:04:54.930645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.930458 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-openshift-ca\") pod \"2709af96-abe6-4467-b9ad-e32c7dd469aa\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " Nov 03 23:04:54.930695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.930679 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-xks-ca\") pod \"2709af96-abe6-4467-b9ad-e32c7dd469aa\" (UID: \"2709af96-abe6-4467-b9ad-e32c7dd469aa\") " Nov 03 23:04:54.930855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.930827 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-openshift-ca" pod "2709af96-abe6-4467-b9ad-e32c7dd469aa" (UID: "2709af96-abe6-4467-b9ad-e32c7dd469aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:54.931119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.931096 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:54.932664 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.932633 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-xks-ca" pod "2709af96-abe6-4467-b9ad-e32c7dd469aa" (UID: "2709af96-abe6-4467-b9ad-e32c7dd469aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:54.932664 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:54.932649 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2709af96-abe6-4467-b9ad-e32c7dd469aa-kube-api-access-7lvj5" pod "2709af96-abe6-4467-b9ad-e32c7dd469aa" (UID: "2709af96-abe6-4467-b9ad-e32c7dd469aa"). InnerVolumeSpecName "kube-api-access-7lvj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:55.032898 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.032865 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709af96-abe6-4467-b9ad-e32c7dd469aa-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:55.032898 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.032893 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lvj5\" (UniqueName: \"kubernetes.io/projected/2709af96-abe6-4467-b9ad-e32c7dd469aa-kube-api-access-7lvj5\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:55.642690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.642661 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" Nov 03 23:04:55.643108 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.642698 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr" event={"ID":"2709af96-abe6-4467-b9ad-e32c7dd469aa","Type":"ContainerDied","Data":"c43c16f135fd92b559ece376d74f6548493d9d27b51ae7201cb9448d15aad974"} Nov 03 23:04:55.643108 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.642744 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c43c16f135fd92b559ece376d74f6548493d9d27b51ae7201cb9448d15aad974" Nov 03 23:04:55.736885 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.736850 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" path="/var/lib/kubelet/pods/2778075f-b2e9-4e82-b8ac-7489c9a55918/volumes" Nov 03 23:04:55.737475 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.737453 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" path="/var/lib/kubelet/pods/d8f3c564-4549-49be-aba0-e474d0ad07aa/volumes" Nov 03 23:04:55.819672 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.819644 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:55.945122 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.945090 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-xks-ca\") pod \"0bf36ae2-a437-4a1e-adc5-b554beb18795\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " Nov 03 23:04:55.945370 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.945350 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-openshift-ca\") pod \"0bf36ae2-a437-4a1e-adc5-b554beb18795\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " Nov 03 23:04:55.945505 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.945488 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf36ae2-a437-4a1e-adc5-b554beb18795-kube-api-access-l8qp8\" (UniqueName: \"kubernetes.io/projected/0bf36ae2-a437-4a1e-adc5-b554beb18795-kube-api-access-l8qp8\") pod \"0bf36ae2-a437-4a1e-adc5-b554beb18795\" (UID: \"0bf36ae2-a437-4a1e-adc5-b554beb18795\") " Nov 03 23:04:55.945732 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.945705 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-openshift-ca" pod "0bf36ae2-a437-4a1e-adc5-b554beb18795" (UID: "0bf36ae2-a437-4a1e-adc5-b554beb18795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:04:55.946017 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.945999 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:55.947627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.947599 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-xks-ca" pod "0bf36ae2-a437-4a1e-adc5-b554beb18795" (UID: "0bf36ae2-a437-4a1e-adc5-b554beb18795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:04:55.947907 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:55.947886 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf36ae2-a437-4a1e-adc5-b554beb18795-kube-api-access-l8qp8" pod "0bf36ae2-a437-4a1e-adc5-b554beb18795" (UID: "0bf36ae2-a437-4a1e-adc5-b554beb18795"). InnerVolumeSpecName "kube-api-access-l8qp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:04:56.047141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:56.047103 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8qp8\" (UniqueName: \"kubernetes.io/projected/0bf36ae2-a437-4a1e-adc5-b554beb18795-kube-api-access-l8qp8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:56.047141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:56.047137 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf36ae2-a437-4a1e-adc5-b554beb18795-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:04:56.651264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:56.651228 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" Nov 03 23:04:56.651264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:56.651248 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq" event={"ID":"0bf36ae2-a437-4a1e-adc5-b554beb18795","Type":"ContainerDied","Data":"cff73936f57bb59480e87537d6cbc8e93a9a2c2b71db95643b854aa88d3d3886"} Nov 03 23:04:56.651726 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:56.651321 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff73936f57bb59480e87537d6cbc8e93a9a2c2b71db95643b854aa88d3d3886" Nov 03 23:04:59.536276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.536242 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz"] Nov 03 23:04:59.537122 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537099 4903 cpu_manager.go:414] "containerMap: removing container" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" Nov 03 23:04:59.537122 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537120 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537138 4903 cpu_manager.go:414] "containerMap: removing container" podUID="13fdf073-9c1e-4695-8ce6-af0287391ef3" containerName="payload-pre-processing" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537143 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fdf073-9c1e-4695-8ce6-af0287391ef3" containerName="payload-pre-processing" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537178 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0bf36ae2-a437-4a1e-adc5-b554beb18795" containerName="revoke-keys" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537183 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf36ae2-a437-4a1e-adc5-b554beb18795" containerName="revoke-keys" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537195 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b2341ba1-d0ba-460a-ab1e-58a6fd17e217" containerName="payload-processing" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537200 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2341ba1-d0ba-460a-ab1e-58a6fd17e217" containerName="payload-processing" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537211 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2709af96-abe6-4467-b9ad-e32c7dd469aa" containerName="revoke-keys" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537216 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2709af96-abe6-4467-b9ad-e32c7dd469aa" containerName="revoke-keys" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537228 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537234 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537343 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf36ae2-a437-4a1e-adc5-b554beb18795" containerName="revoke-keys" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537352 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2341ba1-d0ba-460a-ab1e-58a6fd17e217" containerName="payload-processing" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537360 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2709af96-abe6-4467-b9ad-e32c7dd469aa" containerName="revoke-keys" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537367 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2778075f-b2e9-4e82-b8ac-7489c9a55918" containerName="main" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537373 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f3c564-4549-49be-aba0-e474d0ad07aa" containerName="main" Nov 03 23:04:59.537389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.537379 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fdf073-9c1e-4695-8ce6-af0287391ef3" containerName="payload-pre-processing" Nov 03 23:04:59.540791 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.540771 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.558889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.558862 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz"] Nov 03 23:04:59.564491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.564469 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:04:59.564666 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.564501 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:04:59.564666 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.564525 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:04:59.568270 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.568247 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:04:59.711481 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.711450 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs94k\" (UniqueName: \"kubernetes.io/projected/2e9ee770-79c3-4638-ab08-fc36e583b704-kube-api-access-rs94k\") pod \"maas-api-revoke-sub-b2a359e6-dsvqz\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.711720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.711520 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a359e6-dsvqz\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.711720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.711639 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a359e6-dsvqz\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.813510 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.813426 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs94k\" (UniqueName: \"kubernetes.io/projected/2e9ee770-79c3-4638-ab08-fc36e583b704-kube-api-access-rs94k\") pod \"maas-api-revoke-sub-b2a359e6-dsvqz\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.813698 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.813512 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a359e6-dsvqz\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.813698 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.813587 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a359e6-dsvqz\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.814277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.814253 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a359e6-dsvqz\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.815958 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.815934 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a359e6-dsvqz\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.830912 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.830887 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v"] Nov 03 23:04:59.835630 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.835611 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:04:59.840151 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.840133 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs94k\" (UniqueName: \"kubernetes.io/projected/2e9ee770-79c3-4638-ab08-fc36e583b704-kube-api-access-rs94k\") pod \"maas-api-revoke-sub-b2a359e6-dsvqz\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:04:59.854258 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.854225 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v"] Nov 03 23:04:59.870311 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:04:59.870009 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:05:00.017460 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.017416 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6bc\" (UniqueName: \"kubernetes.io/projected/c9af51ba-4457-4119-b4f2-8de27c8a36b4-kube-api-access-pt6bc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.017679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.017510 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.017679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.017617 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.054452 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.054420 4903 scope.go:122] "RemoveContainer" containerID="ddda87662306b44f7c59eb058c1dc02f45c651e6db6102e876fcbc61ead07e8d" Nov 03 23:05:00.065045 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.065020 4903 scope.go:122] "RemoveContainer" containerID="a91098bbc44d946c23635a17dba9d50da9c0e3af6d6c8a7a6b453463f25d0808" Nov 03 23:05:00.124118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.124083 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.124254 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.124210 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.124855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.124507 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6bc\" (UniqueName: \"kubernetes.io/projected/c9af51ba-4457-4119-b4f2-8de27c8a36b4-kube-api-access-pt6bc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.125168 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.125142 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.126885 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.126861 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.145964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.145938 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6bc\" (UniqueName: \"kubernetes.io/projected/c9af51ba-4457-4119-b4f2-8de27c8a36b4-kube-api-access-pt6bc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.195323 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.195267 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz"] Nov 03 23:05:00.197516 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:00.197476 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9ee770_79c3_4638_ab08_fc36e583b704.slice/crio-743ca0cc539f863cdf07e36d3e9a62e246782494fe398f4b286e2ec0c10bed10 WatchSource:0}: Error finding container 743ca0cc539f863cdf07e36d3e9a62e246782494fe398f4b286e2ec0c10bed10: Status 404 returned error can't find the container with id 743ca0cc539f863cdf07e36d3e9a62e246782494fe398f4b286e2ec0c10bed10 Nov 03 23:05:00.204033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.204012 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:00.488507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.488473 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt"] Nov 03 23:05:00.488699 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.488675 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" containerID="cri-o://0b81b9e163ebdc4205b95c923e53e574116cabd8fc9cb500b44d0749bc98822a" gracePeriod=30 Nov 03 23:05:00.529570 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.529538 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v"] Nov 03 23:05:00.530331 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:00.530287 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9af51ba_4457_4119_b4f2_8de27c8a36b4.slice/crio-ece30ace238e5d25c827c423d5029a19446219038d433bc3cfbc0e554c22c000 WatchSource:0}: Error finding container ece30ace238e5d25c827c423d5029a19446219038d433bc3cfbc0e554c22c000: Status 404 returned error can't find the container with id ece30ace238e5d25c827c423d5029a19446219038d433bc3cfbc0e554c22c000 Nov 03 23:05:00.686635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.686599 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" event={"ID":"c9af51ba-4457-4119-b4f2-8de27c8a36b4","Type":"ContainerStarted","Data":"ece30ace238e5d25c827c423d5029a19446219038d433bc3cfbc0e554c22c000"} Nov 03 23:05:00.688247 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.688220 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" event={"ID":"2e9ee770-79c3-4638-ab08-fc36e583b704","Type":"ContainerStarted","Data":"f02de15c74272a309ea7c7843965c6498d8253dc594835a087864c9d51a3ab02"} Nov 03 23:05:00.688426 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.688265 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" event={"ID":"2e9ee770-79c3-4638-ab08-fc36e583b704","Type":"ContainerStarted","Data":"743ca0cc539f863cdf07e36d3e9a62e246782494fe398f4b286e2ec0c10bed10"} Nov 03 23:05:00.729450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:00.729403 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" podStartSLOduration=1.729388191 podStartE2EDuration="1.729388191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:00.726769524 +0000 UTC m=+3127.130710499" watchObservedRunningTime="2026-11-03 23:05:00.729388191 +0000 UTC m=+3127.133329168" Nov 03 23:05:01.102929 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.102892 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7"] Nov 03 23:05:01.107513 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.107495 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.126629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.126599 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7"] Nov 03 23:05:01.150267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.150233 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238bf880-9272-415d-9925-af02e100f58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.150481 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.150278 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238bf880-9272-415d-9925-af02e100f58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.150481 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.150319 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjkxk\" (UniqueName: \"kubernetes.io/projected/238bf880-9272-415d-9925-af02e100f58e-kube-api-access-qjkxk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.252008 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.251917 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238bf880-9272-415d-9925-af02e100f58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.252008 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.251961 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238bf880-9272-415d-9925-af02e100f58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.252008 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.251981 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjkxk\" (UniqueName: \"kubernetes.io/projected/238bf880-9272-415d-9925-af02e100f58e-kube-api-access-qjkxk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.252626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.252603 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238bf880-9272-415d-9925-af02e100f58e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.254307 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.254268 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238bf880-9272-415d-9925-af02e100f58e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.276132 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.276105 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjkxk\" (UniqueName: \"kubernetes.io/projected/238bf880-9272-415d-9925-af02e100f58e-kube-api-access-qjkxk\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.425098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.425064 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:01.698924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.698887 4903 generic.go:356] "Generic (PLEG): container finished" podID="c9af51ba-4457-4119-b4f2-8de27c8a36b4" containerID="9425ff9bf5af3438227a94e6f671b24527d1c1d64f1d957b0e370c145a6d53ca" exitCode=0 Nov 03 23:05:01.699375 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.698966 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" event={"ID":"c9af51ba-4457-4119-b4f2-8de27c8a36b4","Type":"ContainerDied","Data":"9425ff9bf5af3438227a94e6f671b24527d1c1d64f1d957b0e370c145a6d53ca"} Nov 03 23:05:01.700895 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.700866 4903 generic.go:356] "Generic (PLEG): container finished" podID="2e9ee770-79c3-4638-ab08-fc36e583b704" containerID="f02de15c74272a309ea7c7843965c6498d8253dc594835a087864c9d51a3ab02" exitCode=0 Nov 03 23:05:01.700996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.700927 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" event={"ID":"2e9ee770-79c3-4638-ab08-fc36e583b704","Type":"ContainerDied","Data":"f02de15c74272a309ea7c7843965c6498d8253dc594835a087864c9d51a3ab02"} Nov 03 23:05:01.739913 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:01.739817 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7"] Nov 03 23:05:01.742988 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:01.742952 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238bf880_9272_415d_9925_af02e100f58e.slice/crio-a079771b8807436041a6977496bf8451c66a0fafcd31c47e0f2c516578f35b7c WatchSource:0}: Error finding container a079771b8807436041a6977496bf8451c66a0fafcd31c47e0f2c516578f35b7c: Status 404 returned error can't find the container with id a079771b8807436041a6977496bf8451c66a0fafcd31c47e0f2c516578f35b7c Nov 03 23:05:02.710785 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.710735 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" event={"ID":"238bf880-9272-415d-9925-af02e100f58e","Type":"ContainerStarted","Data":"3d9ff547a3b1ccf365b1ce53259fed772cfba9465a2aca4f5b982e25bb793e1a"} Nov 03 23:05:02.711245 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.710805 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" event={"ID":"238bf880-9272-415d-9925-af02e100f58e","Type":"ContainerStarted","Data":"a079771b8807436041a6977496bf8451c66a0fafcd31c47e0f2c516578f35b7c"} Nov 03 23:05:02.932886 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.932821 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:05:02.962968 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.962935 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:02.971106 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.971058 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" podStartSLOduration=1.971044398 podStartE2EDuration="1.971044398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:02.753632774 +0000 UTC m=+3129.157573782" watchObservedRunningTime="2026-11-03 23:05:02.971044398 +0000 UTC m=+3129.374985372" Nov 03 23:05:02.972852 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.972821 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e9ee770-79c3-4638-ab08-fc36e583b704-kube-api-access-rs94k\" (UniqueName: \"kubernetes.io/projected/2e9ee770-79c3-4638-ab08-fc36e583b704-kube-api-access-rs94k\") pod \"2e9ee770-79c3-4638-ab08-fc36e583b704\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " Nov 03 23:05:02.973352 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.973328 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-openshift-ca\") pod \"2e9ee770-79c3-4638-ab08-fc36e583b704\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " Nov 03 23:05:02.973720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.973680 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-openshift-ca" pod "2e9ee770-79c3-4638-ab08-fc36e583b704" (UID: "2e9ee770-79c3-4638-ab08-fc36e583b704"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:02.973857 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.973782 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-openshift-ca\") pod \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " Nov 03 23:05:02.973857 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.973828 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-xks-ca\") pod \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " Nov 03 23:05:02.974095 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.973861 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-xks-ca\") pod \"2e9ee770-79c3-4638-ab08-fc36e583b704\" (UID: \"2e9ee770-79c3-4638-ab08-fc36e583b704\") " Nov 03 23:05:02.974095 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.973970 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9af51ba-4457-4119-b4f2-8de27c8a36b4-kube-api-access-pt6bc\" (UniqueName: \"kubernetes.io/projected/c9af51ba-4457-4119-b4f2-8de27c8a36b4-kube-api-access-pt6bc\") pod \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\" (UID: \"c9af51ba-4457-4119-b4f2-8de27c8a36b4\") " Nov 03 23:05:02.976463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.974383 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-openshift-ca" pod "c9af51ba-4457-4119-b4f2-8de27c8a36b4" (UID: "c9af51ba-4457-4119-b4f2-8de27c8a36b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:02.976463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.974718 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:02.976463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.974737 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:02.976463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.976099 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e9ee770-79c3-4638-ab08-fc36e583b704-kube-api-access-rs94k" pod "2e9ee770-79c3-4638-ab08-fc36e583b704" (UID: "2e9ee770-79c3-4638-ab08-fc36e583b704"). InnerVolumeSpecName "kube-api-access-rs94k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:02.976743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.976696 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9af51ba-4457-4119-b4f2-8de27c8a36b4-kube-api-access-pt6bc" pod "c9af51ba-4457-4119-b4f2-8de27c8a36b4" (UID: "c9af51ba-4457-4119-b4f2-8de27c8a36b4"). InnerVolumeSpecName "kube-api-access-pt6bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:02.978915 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.978888 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-xks-ca" pod "c9af51ba-4457-4119-b4f2-8de27c8a36b4" (UID: "c9af51ba-4457-4119-b4f2-8de27c8a36b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:02.981432 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:02.981403 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-xks-ca" pod "2e9ee770-79c3-4638-ab08-fc36e583b704" (UID: "2e9ee770-79c3-4638-ab08-fc36e583b704"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:03.076072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.076030 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs94k\" (UniqueName: \"kubernetes.io/projected/2e9ee770-79c3-4638-ab08-fc36e583b704-kube-api-access-rs94k\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:03.076072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.076066 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9af51ba-4457-4119-b4f2-8de27c8a36b4-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:03.076072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.076077 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e9ee770-79c3-4638-ab08-fc36e583b704-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:03.076343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.076085 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt6bc\" (UniqueName: \"kubernetes.io/projected/c9af51ba-4457-4119-b4f2-8de27c8a36b4-kube-api-access-pt6bc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:03.721152 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.721115 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" Nov 03 23:05:03.721615 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.721146 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v" event={"ID":"c9af51ba-4457-4119-b4f2-8de27c8a36b4","Type":"ContainerDied","Data":"ece30ace238e5d25c827c423d5029a19446219038d433bc3cfbc0e554c22c000"} Nov 03 23:05:03.721615 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.721206 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ece30ace238e5d25c827c423d5029a19446219038d433bc3cfbc0e554c22c000" Nov 03 23:05:03.722639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.722619 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" Nov 03 23:05:03.722764 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.722657 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz" event={"ID":"2e9ee770-79c3-4638-ab08-fc36e583b704","Type":"ContainerDied","Data":"743ca0cc539f863cdf07e36d3e9a62e246782494fe398f4b286e2ec0c10bed10"} Nov 03 23:05:03.722764 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.722698 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="743ca0cc539f863cdf07e36d3e9a62e246782494fe398f4b286e2ec0c10bed10" Nov 03 23:05:03.724414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.724387 4903 generic.go:356] "Generic (PLEG): container finished" podID="238bf880-9272-415d-9925-af02e100f58e" containerID="3d9ff547a3b1ccf365b1ce53259fed772cfba9465a2aca4f5b982e25bb793e1a" exitCode=0 Nov 03 23:05:03.724537 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.724469 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" event={"ID":"238bf880-9272-415d-9925-af02e100f58e","Type":"ContainerDied","Data":"3d9ff547a3b1ccf365b1ce53259fed772cfba9465a2aca4f5b982e25bb793e1a"} Nov 03 23:05:03.919506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.919469 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm"] Nov 03 23:05:03.920135 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.920120 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c9af51ba-4457-4119-b4f2-8de27c8a36b4" containerName="revoke-keys" Nov 03 23:05:03.920181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.920137 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9af51ba-4457-4119-b4f2-8de27c8a36b4" containerName="revoke-keys" Nov 03 23:05:03.920181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.920148 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2e9ee770-79c3-4638-ab08-fc36e583b704" containerName="revoke-keys" Nov 03 23:05:03.920181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.920154 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e9ee770-79c3-4638-ab08-fc36e583b704" containerName="revoke-keys" Nov 03 23:05:03.920284 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.920273 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9af51ba-4457-4119-b4f2-8de27c8a36b4" containerName="revoke-keys" Nov 03 23:05:03.920284 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.920282 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e9ee770-79c3-4638-ab08-fc36e583b704" containerName="revoke-keys" Nov 03 23:05:03.924054 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.924038 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:03.941623 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.941591 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm"] Nov 03 23:05:03.988883 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.988797 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:03.988883 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.988845 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:03.989142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:03.989116 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lvn\" (UniqueName: \"kubernetes.io/projected/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-kube-api-access-w4lvn\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:04.090912 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.090873 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lvn\" (UniqueName: \"kubernetes.io/projected/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-kube-api-access-w4lvn\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:04.091113 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.090959 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:04.091113 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.090995 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:04.091779 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.091751 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:04.093541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.093514 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:04.112766 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.112735 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lvn\" (UniqueName: \"kubernetes.io/projected/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-kube-api-access-w4lvn\") pod \"maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:04.185031 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.184984 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" podUID="a7ec5663-6735-4dab-b18d-a3f3b722c2a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:05:04.238566 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.238529 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:04.466823 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.466763 4903 patch_prober.go:29] interesting pod/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:05:04.467028 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.466847 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:05:04.555331 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.555286 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm"] Nov 03 23:05:04.555761 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:04.555738 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66aca6f_7c38_4e68_9de4_b9cc5bb36ee8.slice/crio-d9ee924c8d002a75ee02d9034e72edd7913ab4c1f762304f4c45e7c11e3f9cfd WatchSource:0}: Error finding container d9ee924c8d002a75ee02d9034e72edd7913ab4c1f762304f4c45e7c11e3f9cfd: Status 404 returned error can't find the container with id d9ee924c8d002a75ee02d9034e72edd7913ab4c1f762304f4c45e7c11e3f9cfd Nov 03 23:05:04.735205 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.735166 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" event={"ID":"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8","Type":"ContainerStarted","Data":"d9ee924c8d002a75ee02d9034e72edd7913ab4c1f762304f4c45e7c11e3f9cfd"} Nov 03 23:05:04.888529 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:04.888501 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:05.003664 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.003627 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/238bf880-9272-415d-9925-af02e100f58e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238bf880-9272-415d-9925-af02e100f58e-maas-api-openshift-ca\") pod \"238bf880-9272-415d-9925-af02e100f58e\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " Nov 03 23:05:05.003875 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.003733 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/238bf880-9272-415d-9925-af02e100f58e-kube-api-access-qjkxk\" (UniqueName: \"kubernetes.io/projected/238bf880-9272-415d-9925-af02e100f58e-kube-api-access-qjkxk\") pod \"238bf880-9272-415d-9925-af02e100f58e\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " Nov 03 23:05:05.003875 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.003824 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/238bf880-9272-415d-9925-af02e100f58e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238bf880-9272-415d-9925-af02e100f58e-maas-api-xks-ca\") pod \"238bf880-9272-415d-9925-af02e100f58e\" (UID: \"238bf880-9272-415d-9925-af02e100f58e\") " Nov 03 23:05:05.004111 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.004079 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/238bf880-9272-415d-9925-af02e100f58e-maas-api-openshift-ca" pod "238bf880-9272-415d-9925-af02e100f58e" (UID: "238bf880-9272-415d-9925-af02e100f58e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:05.004548 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.004523 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238bf880-9272-415d-9925-af02e100f58e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.006405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.006339 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/238bf880-9272-415d-9925-af02e100f58e-maas-api-xks-ca" pod "238bf880-9272-415d-9925-af02e100f58e" (UID: "238bf880-9272-415d-9925-af02e100f58e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:05.006533 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.006483 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238bf880-9272-415d-9925-af02e100f58e-kube-api-access-qjkxk" pod "238bf880-9272-415d-9925-af02e100f58e" (UID: "238bf880-9272-415d-9925-af02e100f58e"). InnerVolumeSpecName "kube-api-access-qjkxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:05.106456 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.106412 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjkxk\" (UniqueName: \"kubernetes.io/projected/238bf880-9272-415d-9925-af02e100f58e-kube-api-access-qjkxk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.106456 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.106444 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238bf880-9272-415d-9925-af02e100f58e-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.752845 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.752815 4903 generic.go:356] "Generic (PLEG): container finished" podID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerID="0b81b9e163ebdc4205b95c923e53e574116cabd8fc9cb500b44d0749bc98822a" exitCode=0 Nov 03 23:05:05.753264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.752881 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" event={"ID":"2ea5c865-19c0-4343-b2d5-f34a94171fa8","Type":"ContainerDied","Data":"0b81b9e163ebdc4205b95c923e53e574116cabd8fc9cb500b44d0749bc98822a"} Nov 03 23:05:05.753264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.752953 4903 scope.go:122] "RemoveContainer" containerID="e603f5b193260da999995423ea21b8587b309ab0cff2d08b3afe86e10ad26a75" Nov 03 23:05:05.754859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.754838 4903 generic.go:356] "Generic (PLEG): container finished" podID="b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8" containerID="d66d37842e91fbb1a73d26d9ddec6c66abe94109585a560554f1d35f12224eae" exitCode=0 Nov 03 23:05:05.754975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.754947 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" event={"ID":"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8","Type":"ContainerDied","Data":"d66d37842e91fbb1a73d26d9ddec6c66abe94109585a560554f1d35f12224eae"} Nov 03 23:05:05.756860 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.756838 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" event={"ID":"238bf880-9272-415d-9925-af02e100f58e","Type":"ContainerDied","Data":"a079771b8807436041a6977496bf8451c66a0fafcd31c47e0f2c516578f35b7c"} Nov 03 23:05:05.756990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.756862 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7" Nov 03 23:05:05.756990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.756882 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a079771b8807436041a6977496bf8451c66a0fafcd31c47e0f2c516578f35b7c" Nov 03 23:05:05.759964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.759943 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:05:05.818426 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818333 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-certs\") pod \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " Nov 03 23:05:05.818426 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818417 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-podinfo\") pod \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " Nov 03 23:05:05.818678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818460 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-data\") pod \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " Nov 03 23:05:05.818678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818507 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-socket\") pod \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " Nov 03 23:05:05.818678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818611 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istiod-ca-cert\") pod \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " Nov 03 23:05:05.818678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818638 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-certs" pod "2ea5c865-19c0-4343-b2d5-f34a94171fa8" (UID: "2ea5c865-19c0-4343-b2d5-f34a94171fa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:05.818678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818667 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-token\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-token\") pod \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " Nov 03 23:05:05.818955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818699 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-kube-api-access-fz5lt\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-kube-api-access-fz5lt\") pod \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " Nov 03 23:05:05.818955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818748 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-envoy\") pod \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " Nov 03 23:05:05.818955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818783 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-credential-socket\") pod \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\" (UID: \"2ea5c865-19c0-4343-b2d5-f34a94171fa8\") " Nov 03 23:05:05.818955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818801 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-data" pod "2ea5c865-19c0-4343-b2d5-f34a94171fa8" (UID: "2ea5c865-19c0-4343-b2d5-f34a94171fa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:05.818955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818839 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-socket" pod "2ea5c865-19c0-4343-b2d5-f34a94171fa8" (UID: "2ea5c865-19c0-4343-b2d5-f34a94171fa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:05.819205 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.818984 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istiod-ca-cert" pod "2ea5c865-19c0-4343-b2d5-f34a94171fa8" (UID: "2ea5c865-19c0-4343-b2d5-f34a94171fa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:05.819205 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.819135 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-credential-socket" pod "2ea5c865-19c0-4343-b2d5-f34a94171fa8" (UID: "2ea5c865-19c0-4343-b2d5-f34a94171fa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:05.819527 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.819500 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.819639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.819530 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.819639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.819545 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.819639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.819563 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.819639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.819577 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.820786 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.820761 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-podinfo" pod "2ea5c865-19c0-4343-b2d5-f34a94171fa8" (UID: "2ea5c865-19c0-4343-b2d5-f34a94171fa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:05.820932 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.820913 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-kube-api-access-fz5lt" pod "2ea5c865-19c0-4343-b2d5-f34a94171fa8" (UID: "2ea5c865-19c0-4343-b2d5-f34a94171fa8"). InnerVolumeSpecName "kube-api-access-fz5lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:05.821175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.821152 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-envoy" pod "2ea5c865-19c0-4343-b2d5-f34a94171fa8" (UID: "2ea5c865-19c0-4343-b2d5-f34a94171fa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:05.821235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.821175 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-token" pod "2ea5c865-19c0-4343-b2d5-f34a94171fa8" (UID: "2ea5c865-19c0-4343-b2d5-f34a94171fa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:05.921450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.921406 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.921450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.921442 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.921450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.921456 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz5lt\" (UniqueName: \"kubernetes.io/projected/2ea5c865-19c0-4343-b2d5-f34a94171fa8-kube-api-access-fz5lt\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:05.921707 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:05.921470 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea5c865-19c0-4343-b2d5-f34a94171fa8-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:06.766691 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:06.766662 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" Nov 03 23:05:06.767177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:06.766700 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt" event={"ID":"2ea5c865-19c0-4343-b2d5-f34a94171fa8","Type":"ContainerDied","Data":"cae59d6922e27621c8273e235667b08de69e8c5f95f46f651cdc73a28f38b1ac"} Nov 03 23:05:06.767177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:06.766759 4903 scope.go:122] "RemoveContainer" containerID="0b81b9e163ebdc4205b95c923e53e574116cabd8fc9cb500b44d0749bc98822a" Nov 03 23:05:06.850608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:06.850535 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt"] Nov 03 23:05:06.856265 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:06.856232 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-162d7b-openshift-default-5d877b48fc-l8prt"] Nov 03 23:05:06.940035 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:06.940005 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:07.040696 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.040612 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-kube-api-access-w4lvn\" (UniqueName: \"kubernetes.io/projected/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-kube-api-access-w4lvn\") pod \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " Nov 03 23:05:07.040848 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.040777 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-openshift-ca\") pod \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " Nov 03 23:05:07.040848 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.040828 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-xks-ca\") pod \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\" (UID: \"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8\") " Nov 03 23:05:07.041151 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.041127 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-openshift-ca" pod "b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8" (UID: "b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:07.041321 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.041272 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:07.042825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.042794 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-kube-api-access-w4lvn" pod "b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8" (UID: "b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8"). InnerVolumeSpecName "kube-api-access-w4lvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:07.042949 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.042884 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-xks-ca" pod "b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8" (UID: "b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:07.142405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.142367 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4lvn\" (UniqueName: \"kubernetes.io/projected/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-kube-api-access-w4lvn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:07.142405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.142400 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:07.758565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.758521 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" path="/var/lib/kubelet/pods/2ea5c865-19c0-4343-b2d5-f34a94171fa8/volumes" Nov 03 23:05:07.777327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.777265 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" Nov 03 23:05:07.777327 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.777279 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm" event={"ID":"b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8","Type":"ContainerDied","Data":"d9ee924c8d002a75ee02d9034e72edd7913ab4c1f762304f4c45e7c11e3f9cfd"} Nov 03 23:05:07.777732 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:07.777339 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9ee924c8d002a75ee02d9034e72edd7913ab4c1f762304f4c45e7c11e3f9cfd" Nov 03 23:05:08.553847 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.553788 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 23:05:08.669120 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.669080 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-plugins-config-volume\") pod \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\" (UID: \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\") " Nov 03 23:05:08.669347 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.669329 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-kube-api-access-98qm8\" (UniqueName: \"kubernetes.io/projected/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-kube-api-access-98qm8\") pod \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\" (UID: \"a7ec5663-6735-4dab-b18d-a3f3b722c2a2\") " Nov 03 23:05:08.669505 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.669482 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-plugins-config-volume" pod "a7ec5663-6735-4dab-b18d-a3f3b722c2a2" (UID: "a7ec5663-6735-4dab-b18d-a3f3b722c2a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:08.669858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.669839 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:08.671792 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.671769 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-kube-api-access-98qm8" pod "a7ec5663-6735-4dab-b18d-a3f3b722c2a2" (UID: "a7ec5663-6735-4dab-b18d-a3f3b722c2a2"). InnerVolumeSpecName "kube-api-access-98qm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:08.771482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.771441 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98qm8\" (UniqueName: \"kubernetes.io/projected/a7ec5663-6735-4dab-b18d-a3f3b722c2a2-kube-api-access-98qm8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:08.786138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.786106 4903 generic.go:356] "Generic (PLEG): container finished" podID="a7ec5663-6735-4dab-b18d-a3f3b722c2a2" containerID="eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd" exitCode=0 Nov 03 23:05:08.786595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.786149 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" Nov 03 23:05:08.786595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.786198 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" event={"ID":"a7ec5663-6735-4dab-b18d-a3f3b722c2a2","Type":"ContainerDied","Data":"eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd"} Nov 03 23:05:08.786595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.786258 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2" event={"ID":"a7ec5663-6735-4dab-b18d-a3f3b722c2a2","Type":"ContainerDied","Data":"30b7f0baf37bbb3a96bfb1d2de70f9a21f870537269acb43f17618395d03440a"} Nov 03 23:05:08.786595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.786320 4903 scope.go:122] "RemoveContainer" containerID="eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd" Nov 03 23:05:08.797525 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.797509 4903 scope.go:122] "RemoveContainer" containerID="eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd" Nov 03 23:05:08.797773 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:05:08.797751 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd\": container with ID starting with eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd not found: ID does not exist" containerID="eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd" Nov 03 23:05:08.797842 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.797779 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd"} err="failed to get container status \"eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd\": rpc error: code = NotFound desc = could not find container \"eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd\": container with ID starting with eac1bf9afbb64e359c70d0e705cdcfe3657fbea83cd82e8df4effcdb97b915dd not found: ID does not exist" Nov 03 23:05:08.835871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.835784 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2"] Nov 03 23:05:08.839859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:08.839836 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-553753-b48c58c59-ntcj2"] Nov 03 23:05:09.738174 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.738137 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ec5663-6735-4dab-b18d-a3f3b722c2a2" path="/var/lib/kubelet/pods/a7ec5663-6735-4dab-b18d-a3f3b722c2a2/volumes" Nov 03 23:05:09.901027 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.900988 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8"] Nov 03 23:05:09.901998 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.901974 4903 cpu_manager.go:414] "containerMap: removing container" podUID="238bf880-9272-415d-9925-af02e100f58e" containerName="revoke-keys" Nov 03 23:05:09.901998 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.901997 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="238bf880-9272-415d-9925-af02e100f58e" containerName="revoke-keys" Nov 03 23:05:09.902189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902013 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" Nov 03 23:05:09.902189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902021 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" Nov 03 23:05:09.902189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902031 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" Nov 03 23:05:09.902189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902038 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" Nov 03 23:05:09.902189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902051 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a7ec5663-6735-4dab-b18d-a3f3b722c2a2" containerName="payload-pre-processing" Nov 03 23:05:09.902189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902059 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ec5663-6735-4dab-b18d-a3f3b722c2a2" containerName="payload-pre-processing" Nov 03 23:05:09.902189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902088 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8" containerName="revoke-keys" Nov 03 23:05:09.902189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902097 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8" containerName="revoke-keys" Nov 03 23:05:09.902612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902272 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="238bf880-9272-415d-9925-af02e100f58e" containerName="revoke-keys" Nov 03 23:05:09.902612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902286 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" Nov 03 23:05:09.902612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902316 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea5c865-19c0-4343-b2d5-f34a94171fa8" containerName="istio-proxy" Nov 03 23:05:09.902612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902328 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ec5663-6735-4dab-b18d-a3f3b722c2a2" containerName="payload-pre-processing" Nov 03 23:05:09.902612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.902342 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8" containerName="revoke-keys" Nov 03 23:05:09.907363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.907342 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:09.923367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.923340 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8"] Nov 03 23:05:09.929081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.929056 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:05:09.929081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.929071 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:05:09.929286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.929075 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:05:09.929286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.929156 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:05:09.988235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.988150 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14689fd3-x8lk8\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:09.988495 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.988378 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14689fd3-x8lk8\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:09.988495 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:09.988470 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6kg\" (UniqueName: \"kubernetes.io/projected/cefb864d-6f1a-418b-b77e-7163001cfd19-kube-api-access-ck6kg\") pod \"maas-api-revoke-sub-14689fd3-x8lk8\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:10.090236 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.090198 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14689fd3-x8lk8\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:10.090466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.090273 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6kg\" (UniqueName: \"kubernetes.io/projected/cefb864d-6f1a-418b-b77e-7163001cfd19-kube-api-access-ck6kg\") pod \"maas-api-revoke-sub-14689fd3-x8lk8\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:10.090466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.090437 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14689fd3-x8lk8\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:10.091064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.091046 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14689fd3-x8lk8\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:10.092628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.092609 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14689fd3-x8lk8\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:10.111582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.111559 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6kg\" (UniqueName: \"kubernetes.io/projected/cefb864d-6f1a-418b-b77e-7163001cfd19-kube-api-access-ck6kg\") pod \"maas-api-revoke-sub-14689fd3-x8lk8\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:10.223961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.223928 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:10.525722 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.525639 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g"] Nov 03 23:05:10.532070 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.532043 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8"] Nov 03 23:05:10.537142 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:10.537117 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefb864d_6f1a_418b_b77e_7163001cfd19.slice/crio-49e53c4124d02dd58423d6a187a52bf48acba366720ffd977513cf3695d3b961 WatchSource:0}: Error finding container 49e53c4124d02dd58423d6a187a52bf48acba366720ffd977513cf3695d3b961: Status 404 returned error can't find the container with id 49e53c4124d02dd58423d6a187a52bf48acba366720ffd977513cf3695d3b961 Nov 03 23:05:10.821051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:10.820874 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" event={"ID":"cefb864d-6f1a-418b-b77e-7163001cfd19","Type":"ContainerStarted","Data":"49e53c4124d02dd58423d6a187a52bf48acba366720ffd977513cf3695d3b961"} Nov 03 23:05:11.831106 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:11.831021 4903 generic.go:356] "Generic (PLEG): container finished" podID="cefb864d-6f1a-418b-b77e-7163001cfd19" containerID="4ae2f3d22da3fb86945d1f6720a1d358a89703764ac0a656d65ef666587bb996" exitCode=0 Nov 03 23:05:11.831538 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:11.831112 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" event={"ID":"cefb864d-6f1a-418b-b77e-7163001cfd19","Type":"ContainerDied","Data":"4ae2f3d22da3fb86945d1f6720a1d358a89703764ac0a656d65ef666587bb996"} Nov 03 23:05:12.988939 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:12.988911 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:13.135657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.135561 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-xks-ca\") pod \"cefb864d-6f1a-418b-b77e-7163001cfd19\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " Nov 03 23:05:13.135820 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.135683 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefb864d-6f1a-418b-b77e-7163001cfd19-kube-api-access-ck6kg\" (UniqueName: \"kubernetes.io/projected/cefb864d-6f1a-418b-b77e-7163001cfd19-kube-api-access-ck6kg\") pod \"cefb864d-6f1a-418b-b77e-7163001cfd19\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " Nov 03 23:05:13.135820 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.135796 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-openshift-ca\") pod \"cefb864d-6f1a-418b-b77e-7163001cfd19\" (UID: \"cefb864d-6f1a-418b-b77e-7163001cfd19\") " Nov 03 23:05:13.136171 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.136146 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-openshift-ca" pod "cefb864d-6f1a-418b-b77e-7163001cfd19" (UID: "cefb864d-6f1a-418b-b77e-7163001cfd19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:13.137698 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.137669 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-xks-ca" pod "cefb864d-6f1a-418b-b77e-7163001cfd19" (UID: "cefb864d-6f1a-418b-b77e-7163001cfd19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:13.137796 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.137722 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefb864d-6f1a-418b-b77e-7163001cfd19-kube-api-access-ck6kg" pod "cefb864d-6f1a-418b-b77e-7163001cfd19" (UID: "cefb864d-6f1a-418b-b77e-7163001cfd19"). InnerVolumeSpecName "kube-api-access-ck6kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:13.237350 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.237281 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6kg\" (UniqueName: \"kubernetes.io/projected/cefb864d-6f1a-418b-b77e-7163001cfd19-kube-api-access-ck6kg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:13.237350 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.237340 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:13.237350 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.237352 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefb864d-6f1a-418b-b77e-7163001cfd19-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:13.854870 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.854786 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" event={"ID":"cefb864d-6f1a-418b-b77e-7163001cfd19","Type":"ContainerDied","Data":"49e53c4124d02dd58423d6a187a52bf48acba366720ffd977513cf3695d3b961"} Nov 03 23:05:13.855178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.855153 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e53c4124d02dd58423d6a187a52bf48acba366720ffd977513cf3695d3b961" Nov 03 23:05:13.855448 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:13.855431 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8" Nov 03 23:05:15.030243 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.030206 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6"] Nov 03 23:05:15.040715 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.040685 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e268eb-mmbn6"] Nov 03 23:05:15.526809 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.526756 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" podUID="42016bc2-bc6d-43af-a7df-658b02f4a602" containerName="payload-processing" containerID="cri-o://0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b" gracePeriod=25 Nov 03 23:05:15.758836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.758797 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e5ea2b-b9dc-469e-a43b-8e390dd573c2" path="/var/lib/kubelet/pods/a3e5ea2b-b9dc-469e-a43b-8e390dd573c2/volumes" Nov 03 23:05:15.841668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.841636 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 23:05:15.874125 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.874089 4903 generic.go:356] "Generic (PLEG): container finished" podID="42016bc2-bc6d-43af-a7df-658b02f4a602" containerID="0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b" exitCode=0 Nov 03 23:05:15.874346 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.874135 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" Nov 03 23:05:15.874346 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.874170 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" event={"ID":"42016bc2-bc6d-43af-a7df-658b02f4a602","Type":"ContainerDied","Data":"0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b"} Nov 03 23:05:15.874346 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.874234 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g" event={"ID":"42016bc2-bc6d-43af-a7df-658b02f4a602","Type":"ContainerDied","Data":"fc007ba4e55fa90c8764014b84408f25f73a571b904fac4f5ff8cf52b09d0f67"} Nov 03 23:05:15.874346 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.874267 4903 scope.go:122] "RemoveContainer" containerID="0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b" Nov 03 23:05:15.887225 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.887196 4903 scope.go:122] "RemoveContainer" containerID="0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b" Nov 03 23:05:15.887562 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:05:15.887539 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b\": container with ID starting with 0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b not found: ID does not exist" containerID="0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b" Nov 03 23:05:15.887632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.887569 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b"} err="failed to get container status \"0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b\": rpc error: code = NotFound desc = could not find container \"0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b\": container with ID starting with 0adef085f5a1dcbc48ddeb13b7ffc03eda5e24ca83ff9d1ae26f2043df35bc5b not found: ID does not exist" Nov 03 23:05:15.977919 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.977885 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42016bc2-bc6d-43af-a7df-658b02f4a602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42016bc2-bc6d-43af-a7df-658b02f4a602-plugins-config-volume\") pod \"42016bc2-bc6d-43af-a7df-658b02f4a602\" (UID: \"42016bc2-bc6d-43af-a7df-658b02f4a602\") " Nov 03 23:05:15.978124 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.978020 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42016bc2-bc6d-43af-a7df-658b02f4a602-kube-api-access-9h4pj\" (UniqueName: \"kubernetes.io/projected/42016bc2-bc6d-43af-a7df-658b02f4a602-kube-api-access-9h4pj\") pod \"42016bc2-bc6d-43af-a7df-658b02f4a602\" (UID: \"42016bc2-bc6d-43af-a7df-658b02f4a602\") " Nov 03 23:05:15.978384 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.978355 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42016bc2-bc6d-43af-a7df-658b02f4a602-plugins-config-volume" pod "42016bc2-bc6d-43af-a7df-658b02f4a602" (UID: "42016bc2-bc6d-43af-a7df-658b02f4a602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:15.978620 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.978590 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42016bc2-bc6d-43af-a7df-658b02f4a602-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:15.980375 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:15.980353 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42016bc2-bc6d-43af-a7df-658b02f4a602-kube-api-access-9h4pj" pod "42016bc2-bc6d-43af-a7df-658b02f4a602" (UID: "42016bc2-bc6d-43af-a7df-658b02f4a602"). InnerVolumeSpecName "kube-api-access-9h4pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:16.079913 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:16.079829 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4pj\" (UniqueName: \"kubernetes.io/projected/42016bc2-bc6d-43af-a7df-658b02f4a602-kube-api-access-9h4pj\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:16.220433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:16.220406 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g"] Nov 03 23:05:16.231092 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:16.231058 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-281732-645dfbdbf4-27m7g"] Nov 03 23:05:17.735029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:17.734999 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42016bc2-bc6d-43af-a7df-658b02f4a602" path="/var/lib/kubelet/pods/42016bc2-bc6d-43af-a7df-658b02f4a602/volumes" Nov 03 23:05:18.384570 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:05:18.384535 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:05:19.426530 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:19.426490 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm"] Nov 03 23:05:19.437255 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:19.437216 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4ad693-f2gdm"] Nov 03 23:05:19.734670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:19.734582 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8" path="/var/lib/kubelet/pods/b66aca6f-7c38-4e68-9de4-b9cc5bb36ee8/volumes" Nov 03 23:05:21.609222 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:21.609180 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p"] Nov 03 23:05:21.617787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:21.617752 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-553753-bqw9p"] Nov 03 23:05:21.735762 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:21.735724 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459552e8-0778-4a23-a363-52182defd2da" path="/var/lib/kubelet/pods/459552e8-0778-4a23-a363-52182defd2da/volumes" Nov 03 23:05:24.074667 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:24.074635 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v"] Nov 03 23:05:24.077964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:24.077937 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-281732-j9v5v"] Nov 03 23:05:25.734453 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:25.734401 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9af51ba-4457-4119-b4f2-8de27c8a36b4" path="/var/lib/kubelet/pods/c9af51ba-4457-4119-b4f2-8de27c8a36b4/volumes" Nov 03 23:05:26.840855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:26.840822 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84"] Nov 03 23:05:26.841396 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:26.841047 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" containerID="cri-o://30359d989acb4a41a54c9bb5c162f991c81296355eb6b597c789277d378f4be7" gracePeriod=30 Nov 03 23:05:26.856342 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:26.856312 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s"] Nov 03 23:05:26.856533 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:26.856513 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerName="istio-proxy" containerID="cri-o://fc696d8760a84e0f40c122009e1e76c1fb90bc415fc11383c6043d5f9d686447" gracePeriod=30 Nov 03 23:05:27.393545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.393504 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh"] Nov 03 23:05:27.394407 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.394382 4903 cpu_manager.go:414] "containerMap: removing container" podUID="42016bc2-bc6d-43af-a7df-658b02f4a602" containerName="payload-processing" Nov 03 23:05:27.394407 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.394404 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42016bc2-bc6d-43af-a7df-658b02f4a602" containerName="payload-processing" Nov 03 23:05:27.394575 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.394424 4903 cpu_manager.go:414] "containerMap: removing container" podUID="cefb864d-6f1a-418b-b77e-7163001cfd19" containerName="revoke-keys" Nov 03 23:05:27.394575 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.394429 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefb864d-6f1a-418b-b77e-7163001cfd19" containerName="revoke-keys" Nov 03 23:05:27.394575 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.394551 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="42016bc2-bc6d-43af-a7df-658b02f4a602" containerName="payload-processing" Nov 03 23:05:27.394575 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.394561 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefb864d-6f1a-418b-b77e-7163001cfd19" containerName="revoke-keys" Nov 03 23:05:27.398226 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.398209 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.412604 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.412557 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:05:27.412927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.412909 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:05:27.413088 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.413064 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:05:27.413273 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.413235 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:05:27.419612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.419584 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh"] Nov 03 23:05:27.450882 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.450839 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrhf\" (UniqueName: \"kubernetes.io/projected/aa60ced5-7300-436f-ae74-80a9d3ea98fb-kube-api-access-4zrhf\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.451435 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.451409 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.453105 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.451786 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.554250 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.554215 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.554475 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.554381 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.554545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.554499 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrhf\" (UniqueName: \"kubernetes.io/projected/aa60ced5-7300-436f-ae74-80a9d3ea98fb-kube-api-access-4zrhf\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.555442 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.555385 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.557870 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.557842 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.578105 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.577442 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrhf\" (UniqueName: \"kubernetes.io/projected/aa60ced5-7300-436f-ae74-80a9d3ea98fb-kube-api-access-4zrhf\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:27.715945 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:27.714554 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:28.056971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:28.056941 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh"] Nov 03 23:05:28.057618 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:28.057591 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa60ced5_7300_436f_ae74_80a9d3ea98fb.slice/crio-564ea60d176a48ab8e1ecdc0af7e33c96b3bf22e3736c0ef1c47610e43fb1364 WatchSource:0}: Error finding container 564ea60d176a48ab8e1ecdc0af7e33c96b3bf22e3736c0ef1c47610e43fb1364: Status 404 returned error can't find the container with id 564ea60d176a48ab8e1ecdc0af7e33c96b3bf22e3736c0ef1c47610e43fb1364 Nov 03 23:05:29.050316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:29.047082 4903 generic.go:356] "Generic (PLEG): container finished" podID="aa60ced5-7300-436f-ae74-80a9d3ea98fb" containerID="0acb52f053ff7d3cc13baf809d37e4bb79e523878b268895509d9b8153ced0e9" exitCode=0 Nov 03 23:05:29.050316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:29.047477 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" event={"ID":"aa60ced5-7300-436f-ae74-80a9d3ea98fb","Type":"ContainerDied","Data":"0acb52f053ff7d3cc13baf809d37e4bb79e523878b268895509d9b8153ced0e9"} Nov 03 23:05:29.050316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:29.047545 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" event={"ID":"aa60ced5-7300-436f-ae74-80a9d3ea98fb","Type":"ContainerStarted","Data":"564ea60d176a48ab8e1ecdc0af7e33c96b3bf22e3736c0ef1c47610e43fb1364"} Nov 03 23:05:30.222783 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.222758 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:30.290463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.290418 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-openshift-ca\") pod \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " Nov 03 23:05:30.290645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.290498 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa60ced5-7300-436f-ae74-80a9d3ea98fb-kube-api-access-4zrhf\" (UniqueName: \"kubernetes.io/projected/aa60ced5-7300-436f-ae74-80a9d3ea98fb-kube-api-access-4zrhf\") pod \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " Nov 03 23:05:30.290645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.290637 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-xks-ca\") pod \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\" (UID: \"aa60ced5-7300-436f-ae74-80a9d3ea98fb\") " Nov 03 23:05:30.290844 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.290818 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-openshift-ca" pod "aa60ced5-7300-436f-ae74-80a9d3ea98fb" (UID: "aa60ced5-7300-436f-ae74-80a9d3ea98fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:30.291207 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.291181 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:30.292772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.292746 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-xks-ca" pod "aa60ced5-7300-436f-ae74-80a9d3ea98fb" (UID: "aa60ced5-7300-436f-ae74-80a9d3ea98fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:30.292772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.292759 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa60ced5-7300-436f-ae74-80a9d3ea98fb-kube-api-access-4zrhf" pod "aa60ced5-7300-436f-ae74-80a9d3ea98fb" (UID: "aa60ced5-7300-436f-ae74-80a9d3ea98fb"). InnerVolumeSpecName "kube-api-access-4zrhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:30.392252 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.392171 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa60ced5-7300-436f-ae74-80a9d3ea98fb-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:30.392252 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.392201 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zrhf\" (UniqueName: \"kubernetes.io/projected/aa60ced5-7300-436f-ae74-80a9d3ea98fb-kube-api-access-4zrhf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:30.644950 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.644866 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc"] Nov 03 23:05:30.884269 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.884237 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4"] Nov 03 23:05:30.884865 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.884850 4903 cpu_manager.go:414] "containerMap: removing container" podUID="aa60ced5-7300-436f-ae74-80a9d3ea98fb" containerName="revoke-keys" Nov 03 23:05:30.884865 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.884867 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa60ced5-7300-436f-ae74-80a9d3ea98fb" containerName="revoke-keys" Nov 03 23:05:30.884997 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.884988 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa60ced5-7300-436f-ae74-80a9d3ea98fb" containerName="revoke-keys" Nov 03 23:05:30.888678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.888661 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:30.907181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.907156 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4"] Nov 03 23:05:30.999258 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.999219 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:30.999487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.999282 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:30.999487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:30.999396 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ct77\" (UniqueName: \"kubernetes.io/projected/fd5356f8-b0cb-4914-8f9c-635655dbff89-kube-api-access-9ct77\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:31.064739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.064712 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" Nov 03 23:05:31.064940 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.064754 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh" event={"ID":"aa60ced5-7300-436f-ae74-80a9d3ea98fb","Type":"ContainerDied","Data":"564ea60d176a48ab8e1ecdc0af7e33c96b3bf22e3736c0ef1c47610e43fb1364"} Nov 03 23:05:31.064940 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.064804 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="564ea60d176a48ab8e1ecdc0af7e33c96b3bf22e3736c0ef1c47610e43fb1364" Nov 03 23:05:31.100769 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.100741 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:31.100908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.100789 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:31.100908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.100895 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ct77\" (UniqueName: \"kubernetes.io/projected/fd5356f8-b0cb-4914-8f9c-635655dbff89-kube-api-access-9ct77\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:31.101511 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.101485 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:31.103131 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.103110 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:31.123373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.123351 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ct77\" (UniqueName: \"kubernetes.io/projected/fd5356f8-b0cb-4914-8f9c-635655dbff89-kube-api-access-9ct77\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:31.204630 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.204601 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:31.524592 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:31.524562 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4"] Nov 03 23:05:31.526083 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:31.526056 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5356f8_b0cb_4914_8f9c_635655dbff89.slice/crio-a0021b8821a691428962773de4b25ad831495761466679704e649f2c042f6507 WatchSource:0}: Error finding container a0021b8821a691428962773de4b25ad831495761466679704e649f2c042f6507: Status 404 returned error can't find the container with id a0021b8821a691428962773de4b25ad831495761466679704e649f2c042f6507 Nov 03 23:05:32.087084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.087043 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" event={"ID":"01285782-5e2a-4be2-aaa3-54910fc5d13f","Type":"ContainerDied","Data":"30359d989acb4a41a54c9bb5c162f991c81296355eb6b597c789277d378f4be7"} Nov 03 23:05:32.087246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.087144 4903 generic.go:356] "Generic (PLEG): container finished" podID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerID="30359d989acb4a41a54c9bb5c162f991c81296355eb6b597c789277d378f4be7" exitCode=0 Nov 03 23:05:32.089945 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.089918 4903 generic.go:356] "Generic (PLEG): container finished" podID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerID="fc696d8760a84e0f40c122009e1e76c1fb90bc415fc11383c6043d5f9d686447" exitCode=0 Nov 03 23:05:32.090106 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.089988 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" event={"ID":"1e256488-2716-460e-9a4a-e3b9f75014ea","Type":"ContainerDied","Data":"fc696d8760a84e0f40c122009e1e76c1fb90bc415fc11383c6043d5f9d686447"} Nov 03 23:05:32.092069 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.092037 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" event={"ID":"fd5356f8-b0cb-4914-8f9c-635655dbff89","Type":"ContainerStarted","Data":"d209c0abd8ff213078e0bb031cccfe9f210459780cbbd0d7c92e289f4f8fefba"} Nov 03 23:05:32.092238 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.092073 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" event={"ID":"fd5356f8-b0cb-4914-8f9c-635655dbff89","Type":"ContainerStarted","Data":"a0021b8821a691428962773de4b25ad831495761466679704e649f2c042f6507"} Nov 03 23:05:32.140734 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.140634 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" podStartSLOduration=2.140615203 podStartE2EDuration="2.140615203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:32.138110668 +0000 UTC m=+3158.542051644" watchObservedRunningTime="2026-11-03 23:05:32.140615203 +0000 UTC m=+3158.544556179" Nov 03 23:05:32.157204 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.157178 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:05:32.197612 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.197577 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7"] Nov 03 23:05:32.219254 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.219213 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-b023a2-jg9h7"] Nov 03 23:05:32.220470 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.220442 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-envoy\") pod \"1e256488-2716-460e-9a4a-e3b9f75014ea\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " Nov 03 23:05:32.224775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.224694 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e256488-2716-460e-9a4a-e3b9f75014ea-istiod-ca-cert" pod "1e256488-2716-460e-9a4a-e3b9f75014ea" (UID: "1e256488-2716-460e-9a4a-e3b9f75014ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:32.225739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.224612 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-envoy" pod "1e256488-2716-460e-9a4a-e3b9f75014ea" (UID: "1e256488-2716-460e-9a4a-e3b9f75014ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.225739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.225486 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e256488-2716-460e-9a4a-e3b9f75014ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e256488-2716-460e-9a4a-e3b9f75014ea-istiod-ca-cert\") pod \"1e256488-2716-460e-9a4a-e3b9f75014ea\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " Nov 03 23:05:32.225739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.225539 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-socket\") pod \"1e256488-2716-460e-9a4a-e3b9f75014ea\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " Nov 03 23:05:32.225739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.225572 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-credential-socket\") pod \"1e256488-2716-460e-9a4a-e3b9f75014ea\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " Nov 03 23:05:32.226130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.225843 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-data\") pod \"1e256488-2716-460e-9a4a-e3b9f75014ea\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " Nov 03 23:05:32.226130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.225991 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-credential-socket" pod "1e256488-2716-460e-9a4a-e3b9f75014ea" (UID: "1e256488-2716-460e-9a4a-e3b9f75014ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.226130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226042 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-token\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-token\") pod \"1e256488-2716-460e-9a4a-e3b9f75014ea\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " Nov 03 23:05:32.226130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226073 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-kube-api-access-p7dwq\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-kube-api-access-p7dwq\") pod \"1e256488-2716-460e-9a4a-e3b9f75014ea\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " Nov 03 23:05:32.226369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226211 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-certs\") pod \"1e256488-2716-460e-9a4a-e3b9f75014ea\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " Nov 03 23:05:32.226369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226232 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-data" pod "1e256488-2716-460e-9a4a-e3b9f75014ea" (UID: "1e256488-2716-460e-9a4a-e3b9f75014ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.226369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226249 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-podinfo\") pod \"1e256488-2716-460e-9a4a-e3b9f75014ea\" (UID: \"1e256488-2716-460e-9a4a-e3b9f75014ea\") " Nov 03 23:05:32.226990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226737 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-certs" pod "1e256488-2716-460e-9a4a-e3b9f75014ea" (UID: "1e256488-2716-460e-9a4a-e3b9f75014ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.226990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226844 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.226990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226866 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.226990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226880 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.226990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226895 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e256488-2716-460e-9a4a-e3b9f75014ea-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.226990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.226907 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.232581 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.232550 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-socket" pod "1e256488-2716-460e-9a4a-e3b9f75014ea" (UID: "1e256488-2716-460e-9a4a-e3b9f75014ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.233640 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.233619 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-token" pod "1e256488-2716-460e-9a4a-e3b9f75014ea" (UID: "1e256488-2716-460e-9a4a-e3b9f75014ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.233759 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.233687 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-kube-api-access-p7dwq" pod "1e256488-2716-460e-9a4a-e3b9f75014ea" (UID: "1e256488-2716-460e-9a4a-e3b9f75014ea"). InnerVolumeSpecName "kube-api-access-p7dwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.234934 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.234913 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-podinfo" pod "1e256488-2716-460e-9a4a-e3b9f75014ea" (UID: "1e256488-2716-460e-9a4a-e3b9f75014ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:32.267226 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.267196 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:05:32.350389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.344433 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.350389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.344472 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7dwq\" (UniqueName: \"kubernetes.io/projected/1e256488-2716-460e-9a4a-e3b9f75014ea-kube-api-access-p7dwq\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.350389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.344489 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e256488-2716-460e-9a4a-e3b9f75014ea-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.350389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.344503 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e256488-2716-460e-9a4a-e3b9f75014ea-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.445668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.445632 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-podinfo\") pod \"01285782-5e2a-4be2-aaa3-54910fc5d13f\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " Nov 03 23:05:32.445931 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.445683 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01285782-5e2a-4be2-aaa3-54910fc5d13f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01285782-5e2a-4be2-aaa3-54910fc5d13f-istiod-ca-cert\") pod \"01285782-5e2a-4be2-aaa3-54910fc5d13f\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " Nov 03 23:05:32.445931 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.445728 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-socket\") pod \"01285782-5e2a-4be2-aaa3-54910fc5d13f\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " Nov 03 23:05:32.445931 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.445778 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-certs\") pod \"01285782-5e2a-4be2-aaa3-54910fc5d13f\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " Nov 03 23:05:32.445931 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.445809 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-envoy\") pod \"01285782-5e2a-4be2-aaa3-54910fc5d13f\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " Nov 03 23:05:32.445931 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.445887 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-credential-socket\") pod \"01285782-5e2a-4be2-aaa3-54910fc5d13f\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " Nov 03 23:05:32.446277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.445956 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-kube-api-access-srtj7\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-kube-api-access-srtj7\") pod \"01285782-5e2a-4be2-aaa3-54910fc5d13f\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " Nov 03 23:05:32.446277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446023 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-token\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-token\") pod \"01285782-5e2a-4be2-aaa3-54910fc5d13f\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " Nov 03 23:05:32.446277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446100 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-data\") pod \"01285782-5e2a-4be2-aaa3-54910fc5d13f\" (UID: \"01285782-5e2a-4be2-aaa3-54910fc5d13f\") " Nov 03 23:05:32.446277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446224 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-certs" pod "01285782-5e2a-4be2-aaa3-54910fc5d13f" (UID: "01285782-5e2a-4be2-aaa3-54910fc5d13f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.446277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446243 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-socket" pod "01285782-5e2a-4be2-aaa3-54910fc5d13f" (UID: "01285782-5e2a-4be2-aaa3-54910fc5d13f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.446277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446268 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01285782-5e2a-4be2-aaa3-54910fc5d13f-istiod-ca-cert" pod "01285782-5e2a-4be2-aaa3-54910fc5d13f" (UID: "01285782-5e2a-4be2-aaa3-54910fc5d13f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:32.446648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446500 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-data" pod "01285782-5e2a-4be2-aaa3-54910fc5d13f" (UID: "01285782-5e2a-4be2-aaa3-54910fc5d13f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.446648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446579 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-credential-socket" pod "01285782-5e2a-4be2-aaa3-54910fc5d13f" (UID: "01285782-5e2a-4be2-aaa3-54910fc5d13f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.446735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446694 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.446735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446708 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.446735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446717 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01285782-5e2a-4be2-aaa3-54910fc5d13f-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.446735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446726 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.446735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.446735 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.448710 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.448464 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-envoy" pod "01285782-5e2a-4be2-aaa3-54910fc5d13f" (UID: "01285782-5e2a-4be2-aaa3-54910fc5d13f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:32.448830 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.448727 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-token" pod "01285782-5e2a-4be2-aaa3-54910fc5d13f" (UID: "01285782-5e2a-4be2-aaa3-54910fc5d13f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.449028 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.449003 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-podinfo" pod "01285782-5e2a-4be2-aaa3-54910fc5d13f" (UID: "01285782-5e2a-4be2-aaa3-54910fc5d13f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:05:32.449136 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.449073 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-kube-api-access-srtj7" pod "01285782-5e2a-4be2-aaa3-54910fc5d13f" (UID: "01285782-5e2a-4be2-aaa3-54910fc5d13f"). InnerVolumeSpecName "kube-api-access-srtj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:32.548149 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.548112 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.548149 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.548151 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srtj7\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-kube-api-access-srtj7\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.548601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.548166 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.548601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.548180 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01285782-5e2a-4be2-aaa3-54910fc5d13f-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:32.725986 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.725867 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb"] Nov 03 23:05:32.726665 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.726642 4903 cpu_manager.go:414] "containerMap: removing container" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" Nov 03 23:05:32.726665 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.726665 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" Nov 03 23:05:32.726886 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.726705 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerName="istio-proxy" Nov 03 23:05:32.726886 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.726712 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerName="istio-proxy" Nov 03 23:05:32.727008 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.726924 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" containerName="istio-proxy" Nov 03 23:05:32.727008 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.726947 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" containerName="istio-proxy" Nov 03 23:05:32.737613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.737586 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:32.749493 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.749465 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb"] Nov 03 23:05:32.751589 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.751277 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:32.751589 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.751477 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvwn\" (UniqueName: \"kubernetes.io/projected/7f6a1125-a543-4990-bc84-7123eae4e90d-kube-api-access-2pvwn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:32.751895 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.751599 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:32.852603 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.852562 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:32.852805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.852668 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvwn\" (UniqueName: \"kubernetes.io/projected/7f6a1125-a543-4990-bc84-7123eae4e90d-kube-api-access-2pvwn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:32.852805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.852744 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:32.853214 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.853195 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:32.855232 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.855208 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:32.873662 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:32.873635 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvwn\" (UniqueName: \"kubernetes.io/projected/7f6a1125-a543-4990-bc84-7123eae4e90d-kube-api-access-2pvwn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:33.067963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.067876 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:33.101542 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.101512 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" Nov 03 23:05:33.101691 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.101514 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s" event={"ID":"1e256488-2716-460e-9a4a-e3b9f75014ea","Type":"ContainerDied","Data":"68e99a3eb8c01bac98c6ec3bb8163600cb6625a57d6daf9d233b515259dd51cf"} Nov 03 23:05:33.101691 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.101686 4903 scope.go:122] "RemoveContainer" containerID="fc696d8760a84e0f40c122009e1e76c1fb90bc415fc11383c6043d5f9d686447" Nov 03 23:05:33.103507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.103477 4903 generic.go:356] "Generic (PLEG): container finished" podID="fd5356f8-b0cb-4914-8f9c-635655dbff89" containerID="d209c0abd8ff213078e0bb031cccfe9f210459780cbbd0d7c92e289f4f8fefba" exitCode=0 Nov 03 23:05:33.103642 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.103561 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" event={"ID":"fd5356f8-b0cb-4914-8f9c-635655dbff89","Type":"ContainerDied","Data":"d209c0abd8ff213078e0bb031cccfe9f210459780cbbd0d7c92e289f4f8fefba"} Nov 03 23:05:33.105493 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.105469 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" Nov 03 23:05:33.105595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.105469 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84" event={"ID":"01285782-5e2a-4be2-aaa3-54910fc5d13f","Type":"ContainerDied","Data":"9b3979eb48b7fb5654f950eba1c49768a9ab6780bf91bf6a873fb87f60e4d083"} Nov 03 23:05:33.124790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.124475 4903 scope.go:122] "RemoveContainer" containerID="30359d989acb4a41a54c9bb5c162f991c81296355eb6b597c789277d378f4be7" Nov 03 23:05:33.161485 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.161457 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s"] Nov 03 23:05:33.171417 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.171388 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-2bb6s"] Nov 03 23:05:33.252871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.252838 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84"] Nov 03 23:05:33.256032 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.256008 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4ad693-openshift-default-c44fc7bbf-wkk84"] Nov 03 23:05:33.383968 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.383936 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb"] Nov 03 23:05:33.385549 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:33.385524 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6a1125_a543_4990_bc84_7123eae4e90d.slice/crio-9a1ab6519cfc1fd16e8ce3c816338b793e632c608d7f06c6f595c57a275e7b75 WatchSource:0}: Error finding container 9a1ab6519cfc1fd16e8ce3c816338b793e632c608d7f06c6f595c57a275e7b75: Status 404 returned error can't find the container with id 9a1ab6519cfc1fd16e8ce3c816338b793e632c608d7f06c6f595c57a275e7b75 Nov 03 23:05:33.735222 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.735189 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01285782-5e2a-4be2-aaa3-54910fc5d13f" path="/var/lib/kubelet/pods/01285782-5e2a-4be2-aaa3-54910fc5d13f/volumes" Nov 03 23:05:33.735791 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.735774 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e256488-2716-460e-9a4a-e3b9f75014ea" path="/var/lib/kubelet/pods/1e256488-2716-460e-9a4a-e3b9f75014ea/volumes" Nov 03 23:05:33.736221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:33.736200 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="238bf880-9272-415d-9925-af02e100f58e" path="/var/lib/kubelet/pods/238bf880-9272-415d-9925-af02e100f58e/volumes" Nov 03 23:05:34.115859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.115767 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" event={"ID":"7f6a1125-a543-4990-bc84-7123eae4e90d","Type":"ContainerStarted","Data":"04088a0cef670217266f36b9574d4e74e92344e1da9a6c61ff8192c253d3f283"} Nov 03 23:05:34.115859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.115821 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" event={"ID":"7f6a1125-a543-4990-bc84-7123eae4e90d","Type":"ContainerStarted","Data":"9a1ab6519cfc1fd16e8ce3c816338b793e632c608d7f06c6f595c57a275e7b75"} Nov 03 23:05:34.154926 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.154870 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" podStartSLOduration=2.15485445 podStartE2EDuration="2.15485445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:34.151550295 +0000 UTC m=+3160.555491270" watchObservedRunningTime="2026-11-03 23:05:34.15485445 +0000 UTC m=+3160.558795418" Nov 03 23:05:34.266401 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.266374 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:34.372932 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.372879 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-xks-ca\") pod \"fd5356f8-b0cb-4914-8f9c-635655dbff89\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " Nov 03 23:05:34.373051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.372950 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5356f8-b0cb-4914-8f9c-635655dbff89-kube-api-access-9ct77\" (UniqueName: \"kubernetes.io/projected/fd5356f8-b0cb-4914-8f9c-635655dbff89-kube-api-access-9ct77\") pod \"fd5356f8-b0cb-4914-8f9c-635655dbff89\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " Nov 03 23:05:34.373149 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.373129 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-openshift-ca\") pod \"fd5356f8-b0cb-4914-8f9c-635655dbff89\" (UID: \"fd5356f8-b0cb-4914-8f9c-635655dbff89\") " Nov 03 23:05:34.373484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.373462 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-openshift-ca" pod "fd5356f8-b0cb-4914-8f9c-635655dbff89" (UID: "fd5356f8-b0cb-4914-8f9c-635655dbff89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:34.373618 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.373600 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:34.374881 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.374853 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-xks-ca" pod "fd5356f8-b0cb-4914-8f9c-635655dbff89" (UID: "fd5356f8-b0cb-4914-8f9c-635655dbff89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:34.374978 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.374934 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5356f8-b0cb-4914-8f9c-635655dbff89-kube-api-access-9ct77" pod "fd5356f8-b0cb-4914-8f9c-635655dbff89" (UID: "fd5356f8-b0cb-4914-8f9c-635655dbff89"). InnerVolumeSpecName "kube-api-access-9ct77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:34.474920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.474890 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd5356f8-b0cb-4914-8f9c-635655dbff89-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:34.474920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:34.474919 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ct77\" (UniqueName: \"kubernetes.io/projected/fd5356f8-b0cb-4914-8f9c-635655dbff89-kube-api-access-9ct77\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:35.031360 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:35.031323 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx"] Nov 03 23:05:35.041184 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:35.041155 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9da02c3-nqwgx"] Nov 03 23:05:35.127370 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:35.127330 4903 generic.go:356] "Generic (PLEG): container finished" podID="7f6a1125-a543-4990-bc84-7123eae4e90d" containerID="04088a0cef670217266f36b9574d4e74e92344e1da9a6c61ff8192c253d3f283" exitCode=0 Nov 03 23:05:35.127560 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:35.127415 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" event={"ID":"7f6a1125-a543-4990-bc84-7123eae4e90d","Type":"ContainerDied","Data":"04088a0cef670217266f36b9574d4e74e92344e1da9a6c61ff8192c253d3f283"} Nov 03 23:05:35.129046 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:35.129023 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" event={"ID":"fd5356f8-b0cb-4914-8f9c-635655dbff89","Type":"ContainerDied","Data":"a0021b8821a691428962773de4b25ad831495761466679704e649f2c042f6507"} Nov 03 23:05:35.129183 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:35.129065 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0021b8821a691428962773de4b25ad831495761466679704e649f2c042f6507" Nov 03 23:05:35.129183 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:35.129034 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4" Nov 03 23:05:35.736551 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:35.736513 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e15a33-1307-4365-bc54-4435aa1b57d1" path="/var/lib/kubelet/pods/f2e15a33-1307-4365-bc54-4435aa1b57d1/volumes" Nov 03 23:05:36.304351 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.304322 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:36.398425 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.398337 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-openshift-ca\") pod \"7f6a1125-a543-4990-bc84-7123eae4e90d\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " Nov 03 23:05:36.398425 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.398402 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-xks-ca\") pod \"7f6a1125-a543-4990-bc84-7123eae4e90d\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " Nov 03 23:05:36.398661 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.398457 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6a1125-a543-4990-bc84-7123eae4e90d-kube-api-access-2pvwn\" (UniqueName: \"kubernetes.io/projected/7f6a1125-a543-4990-bc84-7123eae4e90d-kube-api-access-2pvwn\") pod \"7f6a1125-a543-4990-bc84-7123eae4e90d\" (UID: \"7f6a1125-a543-4990-bc84-7123eae4e90d\") " Nov 03 23:05:36.398661 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.398637 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-openshift-ca" pod "7f6a1125-a543-4990-bc84-7123eae4e90d" (UID: "7f6a1125-a543-4990-bc84-7123eae4e90d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:36.399024 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.398994 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:36.400572 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.400541 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-xks-ca" pod "7f6a1125-a543-4990-bc84-7123eae4e90d" (UID: "7f6a1125-a543-4990-bc84-7123eae4e90d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:36.400860 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.400838 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6a1125-a543-4990-bc84-7123eae4e90d-kube-api-access-2pvwn" pod "7f6a1125-a543-4990-bc84-7123eae4e90d" (UID: "7f6a1125-a543-4990-bc84-7123eae4e90d"). InnerVolumeSpecName "kube-api-access-2pvwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:36.500772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.500740 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pvwn\" (UniqueName: \"kubernetes.io/projected/7f6a1125-a543-4990-bc84-7123eae4e90d-kube-api-access-2pvwn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:36.500772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:36.500770 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6a1125-a543-4990-bc84-7123eae4e90d-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:37.146363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:37.146333 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" Nov 03 23:05:37.146556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:37.146368 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb" event={"ID":"7f6a1125-a543-4990-bc84-7123eae4e90d","Type":"ContainerDied","Data":"9a1ab6519cfc1fd16e8ce3c816338b793e632c608d7f06c6f595c57a275e7b75"} Nov 03 23:05:37.146556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:37.146416 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a1ab6519cfc1fd16e8ce3c816338b793e632c608d7f06c6f595c57a275e7b75" Nov 03 23:05:39.735629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.735599 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg"] Nov 03 23:05:39.736218 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.736201 4903 cpu_manager.go:414] "containerMap: removing container" podUID="fd5356f8-b0cb-4914-8f9c-635655dbff89" containerName="revoke-keys" Nov 03 23:05:39.736218 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.736218 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5356f8-b0cb-4914-8f9c-635655dbff89" containerName="revoke-keys" Nov 03 23:05:39.736354 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.736247 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7f6a1125-a543-4990-bc84-7123eae4e90d" containerName="revoke-keys" Nov 03 23:05:39.736354 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.736252 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6a1125-a543-4990-bc84-7123eae4e90d" containerName="revoke-keys" Nov 03 23:05:39.736472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.736379 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5356f8-b0cb-4914-8f9c-635655dbff89" containerName="revoke-keys" Nov 03 23:05:39.736472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.736394 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6a1125-a543-4990-bc84-7123eae4e90d" containerName="revoke-keys" Nov 03 23:05:39.740191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.740170 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:39.750153 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.749921 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg"] Nov 03 23:05:39.754753 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.754729 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:05:39.754888 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.754769 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:05:39.843571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.843535 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4tf6\" (UniqueName: \"kubernetes.io/projected/93c2cfb4-ca6f-427a-a46a-167e20f3d170-kube-api-access-d4tf6\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:39.843752 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.843646 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:39.843852 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.843793 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:39.843925 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.843894 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:05:39.844051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.844035 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:05:39.944946 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.944911 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:39.945104 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.945048 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4tf6\" (UniqueName: \"kubernetes.io/projected/93c2cfb4-ca6f-427a-a46a-167e20f3d170-kube-api-access-d4tf6\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:39.945104 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.945077 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:39.945724 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.945700 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:39.947370 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.947343 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:39.966441 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:39.966416 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4tf6\" (UniqueName: \"kubernetes.io/projected/93c2cfb4-ca6f-427a-a46a-167e20f3d170-kube-api-access-d4tf6\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:40.057949 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:40.057866 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:40.369235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:40.369206 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg"] Nov 03 23:05:40.369821 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:40.369792 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c2cfb4_ca6f_427a_a46a_167e20f3d170.slice/crio-475f70e1154f8484d252e82994e73cdec00e2fd19058dfb89668211a1c8a6698 WatchSource:0}: Error finding container 475f70e1154f8484d252e82994e73cdec00e2fd19058dfb89668211a1c8a6698: Status 404 returned error can't find the container with id 475f70e1154f8484d252e82994e73cdec00e2fd19058dfb89668211a1c8a6698 Nov 03 23:05:41.193082 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:41.193042 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" event={"ID":"93c2cfb4-ca6f-427a-a46a-167e20f3d170","Type":"ContainerStarted","Data":"79abe1ef67277dd83751d0fcc25a67a762155c7762e1a73ba7a32fc38165cbb8"} Nov 03 23:05:41.193526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:41.193108 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" event={"ID":"93c2cfb4-ca6f-427a-a46a-167e20f3d170","Type":"ContainerStarted","Data":"475f70e1154f8484d252e82994e73cdec00e2fd19058dfb89668211a1c8a6698"} Nov 03 23:05:41.230012 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:41.229962 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" podStartSLOduration=2.22994854 podStartE2EDuration="2.22994854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:41.229416878 +0000 UTC m=+3167.633357853" watchObservedRunningTime="2026-11-03 23:05:41.22994854 +0000 UTC m=+3167.633889515" Nov 03 23:05:41.317634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:41.317593 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r"] Nov 03 23:05:42.204875 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:42.204837 4903 generic.go:356] "Generic (PLEG): container finished" podID="93c2cfb4-ca6f-427a-a46a-167e20f3d170" containerID="79abe1ef67277dd83751d0fcc25a67a762155c7762e1a73ba7a32fc38165cbb8" exitCode=0 Nov 03 23:05:42.205315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:42.204923 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" event={"ID":"93c2cfb4-ca6f-427a-a46a-167e20f3d170","Type":"ContainerDied","Data":"79abe1ef67277dd83751d0fcc25a67a762155c7762e1a73ba7a32fc38165cbb8"} Nov 03 23:05:43.135507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.135473 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7"] Nov 03 23:05:43.166176 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.166145 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5"] Nov 03 23:05:43.475011 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.474970 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:43.598098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.598065 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c2cfb4-ca6f-427a-a46a-167e20f3d170-kube-api-access-d4tf6\" (UniqueName: \"kubernetes.io/projected/93c2cfb4-ca6f-427a-a46a-167e20f3d170-kube-api-access-d4tf6\") pod \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " Nov 03 23:05:43.598271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.598125 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-openshift-ca\") pod \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " Nov 03 23:05:43.598403 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.598378 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-xks-ca\") pod \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\" (UID: \"93c2cfb4-ca6f-427a-a46a-167e20f3d170\") " Nov 03 23:05:43.598630 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.598600 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-openshift-ca" pod "93c2cfb4-ca6f-427a-a46a-167e20f3d170" (UID: "93c2cfb4-ca6f-427a-a46a-167e20f3d170"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:43.599001 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.598978 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:43.600394 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.600369 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-xks-ca" pod "93c2cfb4-ca6f-427a-a46a-167e20f3d170" (UID: "93c2cfb4-ca6f-427a-a46a-167e20f3d170"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:43.600487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.600473 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c2cfb4-ca6f-427a-a46a-167e20f3d170-kube-api-access-d4tf6" pod "93c2cfb4-ca6f-427a-a46a-167e20f3d170" (UID: "93c2cfb4-ca6f-427a-a46a-167e20f3d170"). InnerVolumeSpecName "kube-api-access-d4tf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:43.700535 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.700450 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4tf6\" (UniqueName: \"kubernetes.io/projected/93c2cfb4-ca6f-427a-a46a-167e20f3d170-kube-api-access-d4tf6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:43.700535 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:43.700483 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c2cfb4-ca6f-427a-a46a-167e20f3d170-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:44.227096 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:44.227067 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" Nov 03 23:05:44.227286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:44.227108 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg" event={"ID":"93c2cfb4-ca6f-427a-a46a-167e20f3d170","Type":"ContainerDied","Data":"475f70e1154f8484d252e82994e73cdec00e2fd19058dfb89668211a1c8a6698"} Nov 03 23:05:44.227286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:44.227156 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="475f70e1154f8484d252e82994e73cdec00e2fd19058dfb89668211a1c8a6698" Nov 03 23:05:45.665941 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:45.665897 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" podUID="ffe7b53a-4da6-4dba-b7db-805b8673faf4" containerName="main" containerID="cri-o://d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa" gracePeriod=45 Nov 03 23:05:45.941263 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:45.941237 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:05:46.028059 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028027 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-dshm\") pod \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " Nov 03 23:05:46.028245 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028124 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-model-cache\") pod \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " Nov 03 23:05:46.028245 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028185 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe7b53a-4da6-4dba-b7db-805b8673faf4-kube-api-access-s7hln\" (UniqueName: \"kubernetes.io/projected/ffe7b53a-4da6-4dba-b7db-805b8673faf4-kube-api-access-s7hln\") pod \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " Nov 03 23:05:46.028413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028274 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tmp-dir\") pod \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " Nov 03 23:05:46.028413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028397 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-home\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-home\") pod \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " Nov 03 23:05:46.028529 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028438 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-model-cache" pod "ffe7b53a-4da6-4dba-b7db-805b8673faf4" (UID: "ffe7b53a-4da6-4dba-b7db-805b8673faf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:46.028529 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028455 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tls-certs\") pod \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\" (UID: \"ffe7b53a-4da6-4dba-b7db-805b8673faf4\") " Nov 03 23:05:46.028677 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028643 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tmp-dir" pod "ffe7b53a-4da6-4dba-b7db-805b8673faf4" (UID: "ffe7b53a-4da6-4dba-b7db-805b8673faf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:46.028803 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028779 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-home" pod "ffe7b53a-4da6-4dba-b7db-805b8673faf4" (UID: "ffe7b53a-4da6-4dba-b7db-805b8673faf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:46.028995 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.028976 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:46.029064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.029002 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:46.029064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.029016 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:46.030755 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.030683 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql"] Nov 03 23:05:46.030884 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.030849 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tls-certs" pod "ffe7b53a-4da6-4dba-b7db-805b8673faf4" (UID: "ffe7b53a-4da6-4dba-b7db-805b8673faf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:46.030998 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.030929 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-dshm" pod "ffe7b53a-4da6-4dba-b7db-805b8673faf4" (UID: "ffe7b53a-4da6-4dba-b7db-805b8673faf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:05:46.031267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.031241 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe7b53a-4da6-4dba-b7db-805b8673faf4-kube-api-access-s7hln" pod "ffe7b53a-4da6-4dba-b7db-805b8673faf4" (UID: "ffe7b53a-4da6-4dba-b7db-805b8673faf4"). InnerVolumeSpecName "kube-api-access-s7hln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:46.035233 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.035211 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e5214e-k2vql"] Nov 03 23:05:46.130319 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.130252 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffe7b53a-4da6-4dba-b7db-805b8673faf4-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:46.130319 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.130283 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7hln\" (UniqueName: \"kubernetes.io/projected/ffe7b53a-4da6-4dba-b7db-805b8673faf4-kube-api-access-s7hln\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:46.130525 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.130326 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffe7b53a-4da6-4dba-b7db-805b8673faf4-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:46.244256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.244219 4903 generic.go:356] "Generic (PLEG): container finished" podID="ffe7b53a-4da6-4dba-b7db-805b8673faf4" containerID="d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa" exitCode=0 Nov 03 23:05:46.244467 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.244271 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" Nov 03 23:05:46.244467 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.244321 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" event={"ID":"ffe7b53a-4da6-4dba-b7db-805b8673faf4","Type":"ContainerDied","Data":"d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa"} Nov 03 23:05:46.244467 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.244365 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" event={"ID":"ffe7b53a-4da6-4dba-b7db-805b8673faf4","Type":"ContainerDied","Data":"e96392f54a1ad170fe7b6eb0cd2d61b2b250362ac5a757b730da2b1567c56e8a"} Nov 03 23:05:46.244467 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.244388 4903 scope.go:122] "RemoveContainer" containerID="d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa" Nov 03 23:05:46.256153 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.256135 4903 scope.go:122] "RemoveContainer" containerID="d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa" Nov 03 23:05:46.256435 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:05:46.256417 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa\": container with ID starting with d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa not found: ID does not exist" containerID="d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa" Nov 03 23:05:46.256485 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.256444 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa"} err="failed to get container status \"d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa\": rpc error: code = NotFound desc = could not find container \"d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa\": container with ID starting with d9640fc55cd1c5e22efdadd20ffbfbc1f2dec692b84a3bf1ece80b6770ad37aa not found: ID does not exist" Nov 03 23:05:46.291838 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.291809 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc"] Nov 03 23:05:46.295511 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.295485 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc"] Nov 03 23:05:46.318036 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.318002 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" podUID="b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" containerName="payload-pre-processing" containerID="cri-o://d597d25a664cf0d3f1ad9bfe1d49f12080eae47ccf1f2e7dcdcc0ee9aa2b37ca" gracePeriod=25 Nov 03 23:05:46.867519 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:46.867475 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-26f8ba/rate-test-model-26f8ba-kserve-57bc466989-j8lbc" podUID="ffe7b53a-4da6-4dba-b7db-805b8673faf4" containerName="main" probeResult="failure" output="Get \"https://10.132.1.82:8000/ready\": context deadline exceeded" Nov 03 23:05:47.738417 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:47.738381 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b2f6c2-fc47-4d96-9923-488ae8e4f286" path="/var/lib/kubelet/pods/a9b2f6c2-fc47-4d96-9923-488ae8e4f286/volumes" Nov 03 23:05:47.738871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:47.738854 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe7b53a-4da6-4dba-b7db-805b8673faf4" path="/var/lib/kubelet/pods/ffe7b53a-4da6-4dba-b7db-805b8673faf4/volumes" Nov 03 23:05:48.139375 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.135930 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" podUID="b1142a1b-aebb-4d30-af4b-1b5d716459ec" containerName="payload-processing" containerID="cri-o://562db4f092ef27ce6179113125e434689dfd83a6c575a524ed33831e7b53bf26" gracePeriod=25 Nov 03 23:05:48.167157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.167056 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" podUID="990c7438-dadc-4591-88ca-ab96519f2dec" containerName="payload-pre-processing" containerID="cri-o://7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6" gracePeriod=25 Nov 03 23:05:48.299317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.269741 4903 generic.go:356] "Generic (PLEG): container finished" podID="b1142a1b-aebb-4d30-af4b-1b5d716459ec" containerID="562db4f092ef27ce6179113125e434689dfd83a6c575a524ed33831e7b53bf26" exitCode=0 Nov 03 23:05:48.299317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.269870 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" event={"ID":"b1142a1b-aebb-4d30-af4b-1b5d716459ec","Type":"ContainerDied","Data":"562db4f092ef27ce6179113125e434689dfd83a6c575a524ed33831e7b53bf26"} Nov 03 23:05:48.394939 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:05:48.391714 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:05:48.524522 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.524499 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 23:05:48.527961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.527942 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 23:05:48.562187 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.562127 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4"] Nov 03 23:05:48.572723 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.572691 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-2a2eca-cspt4"] Nov 03 23:05:48.672452 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.672418 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990c7438-dadc-4591-88ca-ab96519f2dec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990c7438-dadc-4591-88ca-ab96519f2dec-plugins-config-volume\") pod \"990c7438-dadc-4591-88ca-ab96519f2dec\" (UID: \"990c7438-dadc-4591-88ca-ab96519f2dec\") " Nov 03 23:05:48.672613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.672496 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1142a1b-aebb-4d30-af4b-1b5d716459ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1142a1b-aebb-4d30-af4b-1b5d716459ec-plugins-config-volume\") pod \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\" (UID: \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\") " Nov 03 23:05:48.672613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.672557 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990c7438-dadc-4591-88ca-ab96519f2dec-kube-api-access-ncxqt\" (UniqueName: \"kubernetes.io/projected/990c7438-dadc-4591-88ca-ab96519f2dec-kube-api-access-ncxqt\") pod \"990c7438-dadc-4591-88ca-ab96519f2dec\" (UID: \"990c7438-dadc-4591-88ca-ab96519f2dec\") " Nov 03 23:05:48.672613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.672602 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1142a1b-aebb-4d30-af4b-1b5d716459ec-kube-api-access-ntj28\" (UniqueName: \"kubernetes.io/projected/b1142a1b-aebb-4d30-af4b-1b5d716459ec-kube-api-access-ntj28\") pod \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\" (UID: \"b1142a1b-aebb-4d30-af4b-1b5d716459ec\") " Nov 03 23:05:48.672877 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.672845 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990c7438-dadc-4591-88ca-ab96519f2dec-plugins-config-volume" pod "990c7438-dadc-4591-88ca-ab96519f2dec" (UID: "990c7438-dadc-4591-88ca-ab96519f2dec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:48.672963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.672942 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1142a1b-aebb-4d30-af4b-1b5d716459ec-plugins-config-volume" pod "b1142a1b-aebb-4d30-af4b-1b5d716459ec" (UID: "b1142a1b-aebb-4d30-af4b-1b5d716459ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:48.673026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.673011 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990c7438-dadc-4591-88ca-ab96519f2dec-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:48.674754 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.674728 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990c7438-dadc-4591-88ca-ab96519f2dec-kube-api-access-ncxqt" pod "990c7438-dadc-4591-88ca-ab96519f2dec" (UID: "990c7438-dadc-4591-88ca-ab96519f2dec"). InnerVolumeSpecName "kube-api-access-ncxqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:48.674875 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.674836 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1142a1b-aebb-4d30-af4b-1b5d716459ec-kube-api-access-ntj28" pod "b1142a1b-aebb-4d30-af4b-1b5d716459ec" (UID: "b1142a1b-aebb-4d30-af4b-1b5d716459ec"). InnerVolumeSpecName "kube-api-access-ntj28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:48.773831 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.773801 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1142a1b-aebb-4d30-af4b-1b5d716459ec-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:48.773831 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.773829 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncxqt\" (UniqueName: \"kubernetes.io/projected/990c7438-dadc-4591-88ca-ab96519f2dec-kube-api-access-ncxqt\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:48.774051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:48.773841 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntj28\" (UniqueName: \"kubernetes.io/projected/b1142a1b-aebb-4d30-af4b-1b5d716459ec-kube-api-access-ntj28\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:49.279267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.279228 4903 generic.go:356] "Generic (PLEG): container finished" podID="990c7438-dadc-4591-88ca-ab96519f2dec" containerID="7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6" exitCode=0 Nov 03 23:05:49.279750 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.279278 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" event={"ID":"990c7438-dadc-4591-88ca-ab96519f2dec","Type":"ContainerDied","Data":"7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6"} Nov 03 23:05:49.279750 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.279287 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" Nov 03 23:05:49.279750 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.279349 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5" event={"ID":"990c7438-dadc-4591-88ca-ab96519f2dec","Type":"ContainerDied","Data":"01cf7d124c3c1162d23557b3dd576e8fb5edb7534c0d2eed2f1dfb0f38f49359"} Nov 03 23:05:49.279750 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.279374 4903 scope.go:122] "RemoveContainer" containerID="7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6" Nov 03 23:05:49.281275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.281253 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" event={"ID":"b1142a1b-aebb-4d30-af4b-1b5d716459ec","Type":"ContainerDied","Data":"8bcbd06ce609efeaf2b9f1b2bc7e13da28994eb6f696b773b540f6845b5d381e"} Nov 03 23:05:49.281404 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.281381 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7" Nov 03 23:05:49.292167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.292081 4903 scope.go:122] "RemoveContainer" containerID="7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6" Nov 03 23:05:49.292434 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:05:49.292406 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6\": container with ID starting with 7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6 not found: ID does not exist" containerID="7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6" Nov 03 23:05:49.292526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.292445 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6"} err="failed to get container status \"7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6\": rpc error: code = NotFound desc = could not find container \"7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6\": container with ID starting with 7968c73c5f31bc002f97a8c2b4d1664b2a7fe68a3ef3e52da6cfd2d5ff17a1b6 not found: ID does not exist" Nov 03 23:05:49.292526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.292470 4903 scope.go:122] "RemoveContainer" containerID="562db4f092ef27ce6179113125e434689dfd83a6c575a524ed33831e7b53bf26" Nov 03 23:05:49.330804 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.330775 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5"] Nov 03 23:05:49.340670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.340642 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-04988c-9dcb589c9-xrtn5"] Nov 03 23:05:49.367833 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.367802 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7"] Nov 03 23:05:49.370781 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.370750 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-04988c-97c55b5f4-2jhm7"] Nov 03 23:05:49.734586 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.734557 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990c7438-dadc-4591-88ca-ab96519f2dec" path="/var/lib/kubelet/pods/990c7438-dadc-4591-88ca-ab96519f2dec/volumes" Nov 03 23:05:49.734986 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.734972 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1142a1b-aebb-4d30-af4b-1b5d716459ec" path="/var/lib/kubelet/pods/b1142a1b-aebb-4d30-af4b-1b5d716459ec/volumes" Nov 03 23:05:49.735395 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:49.735381 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5356f8-b0cb-4914-8f9c-635655dbff89" path="/var/lib/kubelet/pods/fd5356f8-b0cb-4914-8f9c-635655dbff89/volumes" Nov 03 23:05:50.344738 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:50.344705 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f"] Nov 03 23:05:51.016344 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:51.016304 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh"] Nov 03 23:05:51.025981 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:51.025952 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6dcc0-fsbwh"] Nov 03 23:05:51.737020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:51.736981 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa60ced5-7300-436f-ae74-80a9d3ea98fb" path="/var/lib/kubelet/pods/aa60ced5-7300-436f-ae74-80a9d3ea98fb/volumes" Nov 03 23:05:54.103073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:54.103031 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" podUID="b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:05:55.345987 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.345936 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" podUID="4fe7a73b-4510-41ab-accb-b1e4b31cc4ab" containerName="payload-pre-processing" containerID="cri-o://67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5" gracePeriod=25 Nov 03 23:05:55.459842 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.459802 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg"] Nov 03 23:05:55.460129 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.460098 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerName="istio-proxy" containerID="cri-o://57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa" gracePeriod=30 Nov 03 23:05:55.655732 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.655702 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 23:05:55.697794 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.697758 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-plugins-config-volume\") pod \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\" (UID: \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\") " Nov 03 23:05:55.699110 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.698422 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-kube-api-access-sfbc6\" (UniqueName: \"kubernetes.io/projected/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-kube-api-access-sfbc6\") pod \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\" (UID: \"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab\") " Nov 03 23:05:55.699495 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.699447 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-plugins-config-volume" pod "4fe7a73b-4510-41ab-accb-b1e4b31cc4ab" (UID: "4fe7a73b-4510-41ab-accb-b1e4b31cc4ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:55.702540 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.702509 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-kube-api-access-sfbc6" pod "4fe7a73b-4510-41ab-accb-b1e4b31cc4ab" (UID: "4fe7a73b-4510-41ab-accb-b1e4b31cc4ab"). InnerVolumeSpecName "kube-api-access-sfbc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:55.800776 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.800738 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfbc6\" (UniqueName: \"kubernetes.io/projected/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-kube-api-access-sfbc6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:55.800776 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:55.800777 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:56.049065 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049025 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n"] Nov 03 23:05:56.049680 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049664 4903 cpu_manager.go:414] "containerMap: removing container" podUID="ffe7b53a-4da6-4dba-b7db-805b8673faf4" containerName="main" Nov 03 23:05:56.049746 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049682 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe7b53a-4da6-4dba-b7db-805b8673faf4" containerName="main" Nov 03 23:05:56.049746 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049701 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4fe7a73b-4510-41ab-accb-b1e4b31cc4ab" containerName="payload-pre-processing" Nov 03 23:05:56.049746 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049706 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe7a73b-4510-41ab-accb-b1e4b31cc4ab" containerName="payload-pre-processing" Nov 03 23:05:56.049746 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049723 4903 cpu_manager.go:414] "containerMap: removing container" podUID="990c7438-dadc-4591-88ca-ab96519f2dec" containerName="payload-pre-processing" Nov 03 23:05:56.049746 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049729 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990c7438-dadc-4591-88ca-ab96519f2dec" containerName="payload-pre-processing" Nov 03 23:05:56.049918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049750 4903 cpu_manager.go:414] "containerMap: removing container" podUID="93c2cfb4-ca6f-427a-a46a-167e20f3d170" containerName="revoke-keys" Nov 03 23:05:56.049918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049758 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c2cfb4-ca6f-427a-a46a-167e20f3d170" containerName="revoke-keys" Nov 03 23:05:56.049918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049764 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b1142a1b-aebb-4d30-af4b-1b5d716459ec" containerName="payload-processing" Nov 03 23:05:56.049918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049777 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1142a1b-aebb-4d30-af4b-1b5d716459ec" containerName="payload-processing" Nov 03 23:05:56.049918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049867 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1142a1b-aebb-4d30-af4b-1b5d716459ec" containerName="payload-processing" Nov 03 23:05:56.049918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049880 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe7b53a-4da6-4dba-b7db-805b8673faf4" containerName="main" Nov 03 23:05:56.049918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049888 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe7a73b-4510-41ab-accb-b1e4b31cc4ab" containerName="payload-pre-processing" Nov 03 23:05:56.049918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049896 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="990c7438-dadc-4591-88ca-ab96519f2dec" containerName="payload-pre-processing" Nov 03 23:05:56.049918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.049903 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c2cfb4-ca6f-427a-a46a-167e20f3d170" containerName="revoke-keys" Nov 03 23:05:56.053038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.053018 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.067011 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.066976 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:05:56.067924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.067449 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:05:56.071208 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.071188 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n"] Nov 03 23:05:56.104184 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.104140 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqd5j\" (UniqueName: \"kubernetes.io/projected/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-kube-api-access-zqd5j\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.104378 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.104330 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.104462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.104444 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.146906 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.146870 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:05:56.147094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.147024 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:05:56.205633 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.205593 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd5j\" (UniqueName: \"kubernetes.io/projected/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-kube-api-access-zqd5j\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.205818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.205683 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.205818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.205735 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.206383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.206361 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.208252 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.208229 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.226967 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.226942 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqd5j\" (UniqueName: \"kubernetes.io/projected/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-kube-api-access-zqd5j\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.369446 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.369366 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:56.406857 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.406827 4903 generic.go:356] "Generic (PLEG): container finished" podID="4fe7a73b-4510-41ab-accb-b1e4b31cc4ab" containerID="67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5" exitCode=0 Nov 03 23:05:56.406998 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.406889 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" Nov 03 23:05:56.406998 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.406904 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" event={"ID":"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab","Type":"ContainerDied","Data":"67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5"} Nov 03 23:05:56.406998 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.406959 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f" event={"ID":"4fe7a73b-4510-41ab-accb-b1e4b31cc4ab","Type":"ContainerDied","Data":"015ff5e4716a2d28f32d3da7f62a8a6822c31632749c181b1aaff97aaa28c45b"} Nov 03 23:05:56.407134 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.406981 4903 scope.go:122] "RemoveContainer" containerID="67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5" Nov 03 23:05:56.419260 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.419239 4903 scope.go:122] "RemoveContainer" containerID="67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5" Nov 03 23:05:56.419572 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:05:56.419546 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5\": container with ID starting with 67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5 not found: ID does not exist" containerID="67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5" Nov 03 23:05:56.419663 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.419584 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5"} err="failed to get container status \"67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5\": rpc error: code = NotFound desc = could not find container \"67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5\": container with ID starting with 67f0ffd50866368c22c876960c84fed771b6f14cf7627fb6bbd94dd95d1281e5 not found: ID does not exist" Nov 03 23:05:56.459677 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.459343 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f"] Nov 03 23:05:56.461665 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.461173 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-9b841d-6447d9f595-qxv2f"] Nov 03 23:05:56.677114 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:56.677089 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n"] Nov 03 23:05:56.679000 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:05:56.678972 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa68a0b_cbd2_4569_b50f_7699a0aa45f1.slice/crio-20a9c4fac01c494b4930734efa29968f78c72c8185a2d02d01c04de66e8c6d9d WatchSource:0}: Error finding container 20a9c4fac01c494b4930734efa29968f78c72c8185a2d02d01c04de66e8c6d9d: Status 404 returned error can't find the container with id 20a9c4fac01c494b4930734efa29968f78c72c8185a2d02d01c04de66e8c6d9d Nov 03 23:05:57.110755 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:57.110717 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:05:57.431357 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:57.425844 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" event={"ID":"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1","Type":"ContainerStarted","Data":"8a3fc8812e213d11baa410fcca57840e9eb94beab799a9568f3ef3455a08d28c"} Nov 03 23:05:57.431357 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:57.425948 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" event={"ID":"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1","Type":"ContainerStarted","Data":"20a9c4fac01c494b4930734efa29968f78c72c8185a2d02d01c04de66e8c6d9d"} Nov 03 23:05:57.467216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:57.467150 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" podStartSLOduration=1.467136712 podStartE2EDuration="1.467136712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:05:57.463907756 +0000 UTC m=+3183.867848733" watchObservedRunningTime="2026-11-03 23:05:57.467136712 +0000 UTC m=+3183.871077686" Nov 03 23:05:57.737147 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:57.737054 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe7a73b-4510-41ab-accb-b1e4b31cc4ab" path="/var/lib/kubelet/pods/4fe7a73b-4510-41ab-accb-b1e4b31cc4ab/volumes" Nov 03 23:05:57.952148 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:57.952097 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg"] Nov 03 23:05:57.955466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:57.955426 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9b841d-lkqqg"] Nov 03 23:05:58.438889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:58.438854 4903 generic.go:356] "Generic (PLEG): container finished" podID="0fa68a0b-cbd2-4569-b50f-7699a0aa45f1" containerID="8a3fc8812e213d11baa410fcca57840e9eb94beab799a9568f3ef3455a08d28c" exitCode=0 Nov 03 23:05:58.439285 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:58.438938 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" event={"ID":"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1","Type":"ContainerDied","Data":"8a3fc8812e213d11baa410fcca57840e9eb94beab799a9568f3ef3455a08d28c"} Nov 03 23:05:58.607917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:58.607581 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb"] Nov 03 23:05:58.611304 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:58.611266 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-04988c-fmsnb"] Nov 03 23:05:59.629707 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.629682 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:05:59.737735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.737649 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6a1125-a543-4990-bc84-7123eae4e90d" path="/var/lib/kubelet/pods/7f6a1125-a543-4990-bc84-7123eae4e90d/volumes" Nov 03 23:05:59.738457 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.738435 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c2cfb4-ca6f-427a-a46a-167e20f3d170" path="/var/lib/kubelet/pods/93c2cfb4-ca6f-427a-a46a-167e20f3d170/volumes" Nov 03 23:05:59.756068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.756028 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-openshift-ca\") pod \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " Nov 03 23:05:59.756271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.756179 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-kube-api-access-zqd5j\" (UniqueName: \"kubernetes.io/projected/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-kube-api-access-zqd5j\") pod \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " Nov 03 23:05:59.756271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.756215 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-xks-ca\") pod \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\" (UID: \"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1\") " Nov 03 23:05:59.767906 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.767874 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-openshift-ca" pod "0fa68a0b-cbd2-4569-b50f-7699a0aa45f1" (UID: "0fa68a0b-cbd2-4569-b50f-7699a0aa45f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:05:59.770440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.770409 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-xks-ca" pod "0fa68a0b-cbd2-4569-b50f-7699a0aa45f1" (UID: "0fa68a0b-cbd2-4569-b50f-7699a0aa45f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:05:59.770661 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.770638 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-kube-api-access-zqd5j" pod "0fa68a0b-cbd2-4569-b50f-7699a0aa45f1" (UID: "0fa68a0b-cbd2-4569-b50f-7699a0aa45f1"). InnerVolumeSpecName "kube-api-access-zqd5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:05:59.858573 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.858255 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:59.858573 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.858287 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqd5j\" (UniqueName: \"kubernetes.io/projected/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-kube-api-access-zqd5j\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:05:59.858573 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:05:59.858321 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:00.144197 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.144102 4903 scope.go:122] "RemoveContainer" containerID="de0b91cd23ff641b2da34d5ed61d2d97ee9ef1bb02b8fca31f361308fbccf3de" Nov 03 23:06:00.154917 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.154896 4903 scope.go:122] "RemoveContainer" containerID="076e3665b8ed4fcfb8f3b75ee97d0dc14fe0726d05c3bf9d94217c44a3ae868a" Nov 03 23:06:00.164970 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.164952 4903 scope.go:122] "RemoveContainer" containerID="96ccb43185b76758ee0715cfab23bc2d9ac8e0418044f33ad531fe22d6a3648b" Nov 03 23:06:00.174809 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.174787 4903 scope.go:122] "RemoveContainer" containerID="7b9a3e393d37f90693ec0f94f53d0fa074db86cdf37033fe0fedbccdd8514fc6" Nov 03 23:06:00.184161 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.184136 4903 scope.go:122] "RemoveContainer" containerID="3048e80298a1b5ce7e5ce93a192220f0ab8831658a622707fb287ca07186fd92" Nov 03 23:06:00.193979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.193958 4903 scope.go:122] "RemoveContainer" containerID="00d10e790fe48223f1708792231c1b53f48c273079f2adec3988e0cb3f0ab1f3" Nov 03 23:06:00.204237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.204218 4903 scope.go:122] "RemoveContainer" containerID="fa56a9900bb90adba04d4bc32772016b8ff5ee2c330139569d3fd6c5dcf2d14b" Nov 03 23:06:00.228639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.228615 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq"] Nov 03 23:06:00.229240 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.229221 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0fa68a0b-cbd2-4569-b50f-7699a0aa45f1" containerName="revoke-keys" Nov 03 23:06:00.229557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.229241 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa68a0b-cbd2-4569-b50f-7699a0aa45f1" containerName="revoke-keys" Nov 03 23:06:00.229557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.229437 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa68a0b-cbd2-4569-b50f-7699a0aa45f1" containerName="revoke-keys" Nov 03 23:06:00.234811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.234783 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.257080 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.257054 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq"] Nov 03 23:06:00.378701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.378664 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppln\" (UniqueName: \"kubernetes.io/projected/e1c20485-a969-4a59-a475-1dd186547b7b-kube-api-access-5ppln\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.378853 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.378711 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.378853 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.378822 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.462653 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.462624 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" Nov 03 23:06:00.462653 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.462649 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n" event={"ID":"0fa68a0b-cbd2-4569-b50f-7699a0aa45f1","Type":"ContainerDied","Data":"20a9c4fac01c494b4930734efa29968f78c72c8185a2d02d01c04de66e8c6d9d"} Nov 03 23:06:00.462915 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.462701 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20a9c4fac01c494b4930734efa29968f78c72c8185a2d02d01c04de66e8c6d9d" Nov 03 23:06:00.480503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.480457 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppln\" (UniqueName: \"kubernetes.io/projected/e1c20485-a969-4a59-a475-1dd186547b7b-kube-api-access-5ppln\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.480503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.480516 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.480775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.480566 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.481562 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.481539 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.483207 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.483183 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.502010 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.501981 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppln\" (UniqueName: \"kubernetes.io/projected/e1c20485-a969-4a59-a475-1dd186547b7b-kube-api-access-5ppln\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.561098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.561068 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:00.807304 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.807214 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp"] Nov 03 23:06:00.812695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.812676 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:00.831808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.831780 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp"] Nov 03 23:06:00.870807 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.870776 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq"] Nov 03 23:06:00.877716 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:00.877672 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c20485_a969_4a59_a475_1dd186547b7b.slice/crio-ca3ff99ac11c4e1557bf106ad89d4a49c90e4b55a506b0b13f00cdc13315fb2b WatchSource:0}: Error finding container ca3ff99ac11c4e1557bf106ad89d4a49c90e4b55a506b0b13f00cdc13315fb2b: Status 404 returned error can't find the container with id ca3ff99ac11c4e1557bf106ad89d4a49c90e4b55a506b0b13f00cdc13315fb2b Nov 03 23:06:00.886385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.886357 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce90165-vvgkp\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:00.886525 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.886496 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgjfp\" (UniqueName: \"kubernetes.io/projected/6edc41c7-1c97-4f3d-98b1-5182d9be168b-kube-api-access-vgjfp\") pod \"maas-api-revoke-sub-5ce90165-vvgkp\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:00.886688 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.886668 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce90165-vvgkp\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:00.987845 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.987818 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce90165-vvgkp\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:00.987931 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.987893 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgjfp\" (UniqueName: \"kubernetes.io/projected/6edc41c7-1c97-4f3d-98b1-5182d9be168b-kube-api-access-vgjfp\") pod \"maas-api-revoke-sub-5ce90165-vvgkp\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:00.988009 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.987998 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce90165-vvgkp\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:00.988728 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.988699 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce90165-vvgkp\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:00.990444 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:00.990423 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce90165-vvgkp\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:01.009657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.009620 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgjfp\" (UniqueName: \"kubernetes.io/projected/6edc41c7-1c97-4f3d-98b1-5182d9be168b-kube-api-access-vgjfp\") pod \"maas-api-revoke-sub-5ce90165-vvgkp\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:01.123064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.123034 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 23:06:01.133002 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.132974 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:01.291695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.291663 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-certs\") pod \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " Nov 03 23:06:01.291874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.291715 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-envoy\") pod \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " Nov 03 23:06:01.291874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.291767 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istiod-ca-cert\") pod \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " Nov 03 23:06:01.291874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.291850 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-token\") pod \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " Nov 03 23:06:01.292052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.291877 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-data\") pod \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " Nov 03 23:06:01.292052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.291969 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-credential-socket\") pod \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " Nov 03 23:06:01.292052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.291998 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-podinfo\") pod \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " Nov 03 23:06:01.292052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292001 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-certs" pod "35a9ee84-e898-4b08-8889-7f7c6c566ac0" (UID: "35a9ee84-e898-4b08-8889-7f7c6c566ac0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:01.292052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292049 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-socket\") pod \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " Nov 03 23:06:01.292337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292159 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-kube-api-access-6swdr\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-kube-api-access-6swdr\") pod \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\" (UID: \"35a9ee84-e898-4b08-8889-7f7c6c566ac0\") " Nov 03 23:06:01.292337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292228 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-data" pod "35a9ee84-e898-4b08-8889-7f7c6c566ac0" (UID: "35a9ee84-e898-4b08-8889-7f7c6c566ac0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:01.292337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292251 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-credential-socket" pod "35a9ee84-e898-4b08-8889-7f7c6c566ac0" (UID: "35a9ee84-e898-4b08-8889-7f7c6c566ac0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:01.292337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292282 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istiod-ca-cert" pod "35a9ee84-e898-4b08-8889-7f7c6c566ac0" (UID: "35a9ee84-e898-4b08-8889-7f7c6c566ac0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:01.292550 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292484 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-socket" pod "35a9ee84-e898-4b08-8889-7f7c6c566ac0" (UID: "35a9ee84-e898-4b08-8889-7f7c6c566ac0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:01.292822 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292802 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.292822 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292820 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.292977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292829 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.292977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292837 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.292977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.292847 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.294246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.294224 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-token" pod "35a9ee84-e898-4b08-8889-7f7c6c566ac0" (UID: "35a9ee84-e898-4b08-8889-7f7c6c566ac0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:01.294246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.294232 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-podinfo" pod "35a9ee84-e898-4b08-8889-7f7c6c566ac0" (UID: "35a9ee84-e898-4b08-8889-7f7c6c566ac0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:01.294451 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.294436 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-envoy" pod "35a9ee84-e898-4b08-8889-7f7c6c566ac0" (UID: "35a9ee84-e898-4b08-8889-7f7c6c566ac0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:01.294597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.294585 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-kube-api-access-6swdr" pod "35a9ee84-e898-4b08-8889-7f7c6c566ac0" (UID: "35a9ee84-e898-4b08-8889-7f7c6c566ac0"). InnerVolumeSpecName "kube-api-access-6swdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:01.393890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.393806 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.393890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.393835 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.393890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.393845 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6swdr\" (UniqueName: \"kubernetes.io/projected/35a9ee84-e898-4b08-8889-7f7c6c566ac0-kube-api-access-6swdr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.393890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.393853 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a9ee84-e898-4b08-8889-7f7c6c566ac0-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:01.443370 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:01.443330 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edc41c7_1c97_4f3d_98b1_5182d9be168b.slice/crio-cd43a0f8ab1459f23cb3f06861b2b2a8083e350709c0ba2cd5dcf80f2917189c WatchSource:0}: Error finding container cd43a0f8ab1459f23cb3f06861b2b2a8083e350709c0ba2cd5dcf80f2917189c: Status 404 returned error can't find the container with id cd43a0f8ab1459f23cb3f06861b2b2a8083e350709c0ba2cd5dcf80f2917189c Nov 03 23:06:01.443718 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.443691 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp"] Nov 03 23:06:01.472801 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.472767 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" event={"ID":"6edc41c7-1c97-4f3d-98b1-5182d9be168b","Type":"ContainerStarted","Data":"cd43a0f8ab1459f23cb3f06861b2b2a8083e350709c0ba2cd5dcf80f2917189c"} Nov 03 23:06:01.474377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.474349 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" event={"ID":"e1c20485-a969-4a59-a475-1dd186547b7b","Type":"ContainerStarted","Data":"78e6669fdb5b4e5070082243b21f7418915d15e3b4a6e820898ef4e9008cb426"} Nov 03 23:06:01.474497 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.474396 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" event={"ID":"e1c20485-a969-4a59-a475-1dd186547b7b","Type":"ContainerStarted","Data":"ca3ff99ac11c4e1557bf106ad89d4a49c90e4b55a506b0b13f00cdc13315fb2b"} Nov 03 23:06:01.475770 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.475750 4903 generic.go:356] "Generic (PLEG): container finished" podID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerID="57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa" exitCode=0 Nov 03 23:06:01.475849 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.475784 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" Nov 03 23:06:01.475849 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.475827 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" event={"ID":"35a9ee84-e898-4b08-8889-7f7c6c566ac0","Type":"ContainerDied","Data":"57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa"} Nov 03 23:06:01.475935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.475876 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg" event={"ID":"35a9ee84-e898-4b08-8889-7f7c6c566ac0","Type":"ContainerDied","Data":"a3d5278a422cce96b805f510c1ae3bc185c12e5d62e083607b55d3e1bf1e79e3"} Nov 03 23:06:01.475935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.475905 4903 scope.go:122] "RemoveContainer" containerID="57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa" Nov 03 23:06:01.487721 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.487696 4903 scope.go:122] "RemoveContainer" containerID="57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa" Nov 03 23:06:01.488025 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:01.488005 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa\": container with ID starting with 57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa not found: ID does not exist" containerID="57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa" Nov 03 23:06:01.488101 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.488033 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa"} err="failed to get container status \"57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa\": rpc error: code = NotFound desc = could not find container \"57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa\": container with ID starting with 57e54010f5eff1169844b60e16588187a32d5d7340c12637fdce7d10685134aa not found: ID does not exist" Nov 03 23:06:01.517582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.517528 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" podStartSLOduration=1.517513165 podStartE2EDuration="1.517513165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:01.515226849 +0000 UTC m=+3187.919167816" watchObservedRunningTime="2026-11-03 23:06:01.517513165 +0000 UTC m=+3187.921454136" Nov 03 23:06:01.556891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.556861 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg"] Nov 03 23:06:01.560479 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.560449 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-2a2eca-openshift-default-5dd76969dc-wbswg"] Nov 03 23:06:01.736854 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:01.736820 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" path="/var/lib/kubelet/pods/35a9ee84-e898-4b08-8889-7f7c6c566ac0/volumes" Nov 03 23:06:02.484582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:02.484541 4903 generic.go:356] "Generic (PLEG): container finished" podID="e1c20485-a969-4a59-a475-1dd186547b7b" containerID="78e6669fdb5b4e5070082243b21f7418915d15e3b4a6e820898ef4e9008cb426" exitCode=0 Nov 03 23:06:02.485068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:02.484620 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" event={"ID":"e1c20485-a969-4a59-a475-1dd186547b7b","Type":"ContainerDied","Data":"78e6669fdb5b4e5070082243b21f7418915d15e3b4a6e820898ef4e9008cb426"} Nov 03 23:06:02.487406 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:02.487379 4903 generic.go:356] "Generic (PLEG): container finished" podID="6edc41c7-1c97-4f3d-98b1-5182d9be168b" containerID="5002fcaac59c7c81d1ef7fec14e53bef86852a85a4ceb27db905ba8e7f7e7059" exitCode=0 Nov 03 23:06:02.487534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:02.487459 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" event={"ID":"6edc41c7-1c97-4f3d-98b1-5182d9be168b","Type":"ContainerDied","Data":"5002fcaac59c7c81d1ef7fec14e53bef86852a85a4ceb27db905ba8e7f7e7059"} Nov 03 23:06:03.660704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.660669 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:03.694365 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.694343 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:03.827779 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.827690 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-xks-ca\") pod \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " Nov 03 23:06:03.829611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.829578 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-xks-ca\") pod \"e1c20485-a969-4a59-a475-1dd186547b7b\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " Nov 03 23:06:03.829760 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.829644 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c20485-a969-4a59-a475-1dd186547b7b-kube-api-access-5ppln\" (UniqueName: \"kubernetes.io/projected/e1c20485-a969-4a59-a475-1dd186547b7b-kube-api-access-5ppln\") pod \"e1c20485-a969-4a59-a475-1dd186547b7b\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " Nov 03 23:06:03.829760 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.829717 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edc41c7-1c97-4f3d-98b1-5182d9be168b-kube-api-access-vgjfp\" (UniqueName: \"kubernetes.io/projected/6edc41c7-1c97-4f3d-98b1-5182d9be168b-kube-api-access-vgjfp\") pod \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " Nov 03 23:06:03.829919 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.829764 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-openshift-ca\") pod \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\" (UID: \"6edc41c7-1c97-4f3d-98b1-5182d9be168b\") " Nov 03 23:06:03.829919 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.829803 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-openshift-ca\") pod \"e1c20485-a969-4a59-a475-1dd186547b7b\" (UID: \"e1c20485-a969-4a59-a475-1dd186547b7b\") " Nov 03 23:06:03.831665 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.831633 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-openshift-ca" pod "6edc41c7-1c97-4f3d-98b1-5182d9be168b" (UID: "6edc41c7-1c97-4f3d-98b1-5182d9be168b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:03.831804 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.831636 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-openshift-ca" pod "e1c20485-a969-4a59-a475-1dd186547b7b" (UID: "e1c20485-a969-4a59-a475-1dd186547b7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:03.832361 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.832321 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-xks-ca" pod "6edc41c7-1c97-4f3d-98b1-5182d9be168b" (UID: "6edc41c7-1c97-4f3d-98b1-5182d9be168b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:03.833684 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.833657 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c20485-a969-4a59-a475-1dd186547b7b-kube-api-access-5ppln" pod "e1c20485-a969-4a59-a475-1dd186547b7b" (UID: "e1c20485-a969-4a59-a475-1dd186547b7b"). InnerVolumeSpecName "kube-api-access-5ppln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:03.833792 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.833766 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edc41c7-1c97-4f3d-98b1-5182d9be168b-kube-api-access-vgjfp" pod "6edc41c7-1c97-4f3d-98b1-5182d9be168b" (UID: "6edc41c7-1c97-4f3d-98b1-5182d9be168b"). InnerVolumeSpecName "kube-api-access-vgjfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:03.834186 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.834159 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-xks-ca" pod "e1c20485-a969-4a59-a475-1dd186547b7b" (UID: "e1c20485-a969-4a59-a475-1dd186547b7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:03.931507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.931474 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:03.931507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.931501 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:03.931507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.931511 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ppln\" (UniqueName: \"kubernetes.io/projected/e1c20485-a969-4a59-a475-1dd186547b7b-kube-api-access-5ppln\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:03.931761 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.931523 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgjfp\" (UniqueName: \"kubernetes.io/projected/6edc41c7-1c97-4f3d-98b1-5182d9be168b-kube-api-access-vgjfp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:03.931761 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.931533 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edc41c7-1c97-4f3d-98b1-5182d9be168b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:03.931761 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:03.931540 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1c20485-a969-4a59-a475-1dd186547b7b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:04.102730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:04.102643 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" podUID="b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:04.503217 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:04.503183 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" Nov 03 23:06:04.503456 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:04.503217 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp" event={"ID":"6edc41c7-1c97-4f3d-98b1-5182d9be168b","Type":"ContainerDied","Data":"cd43a0f8ab1459f23cb3f06861b2b2a8083e350709c0ba2cd5dcf80f2917189c"} Nov 03 23:06:04.503456 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:04.503282 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd43a0f8ab1459f23cb3f06861b2b2a8083e350709c0ba2cd5dcf80f2917189c" Nov 03 23:06:04.504784 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:04.504767 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" Nov 03 23:06:04.504888 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:04.504817 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq" event={"ID":"e1c20485-a969-4a59-a475-1dd186547b7b","Type":"ContainerDied","Data":"ca3ff99ac11c4e1557bf106ad89d4a49c90e4b55a506b0b13f00cdc13315fb2b"} Nov 03 23:06:04.504888 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:04.504865 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3ff99ac11c4e1557bf106ad89d4a49c90e4b55a506b0b13f00cdc13315fb2b" Nov 03 23:06:06.520023 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.519994 4903 generic.go:356] "Generic (PLEG): container finished" podID="b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" containerID="d597d25a664cf0d3f1ad9bfe1d49f12080eae47ccf1f2e7dcdcc0ee9aa2b37ca" exitCode=0 Nov 03 23:06:06.520472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.520070 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" event={"ID":"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d","Type":"ContainerDied","Data":"d597d25a664cf0d3f1ad9bfe1d49f12080eae47ccf1f2e7dcdcc0ee9aa2b37ca"} Nov 03 23:06:06.611556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.611315 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk"] Nov 03 23:06:06.631964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.631920 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 23:06:06.638571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.638543 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl"] Nov 03 23:06:06.768279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.768246 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-plugins-config-volume\") pod \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\" (UID: \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\") " Nov 03 23:06:06.768499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.768335 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-kube-api-access-7xsqz\" (UniqueName: \"kubernetes.io/projected/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-kube-api-access-7xsqz\") pod \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\" (UID: \"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d\") " Nov 03 23:06:06.768712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.768685 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-plugins-config-volume" pod "b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" (UID: "b8744347-6c12-42c5-b7b3-9e5f1ebabd0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:06.770554 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.770531 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-kube-api-access-7xsqz" pod "b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" (UID: "b8744347-6c12-42c5-b7b3-9e5f1ebabd0d"). InnerVolumeSpecName "kube-api-access-7xsqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:06.870545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.870459 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:06.870545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:06.870493 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xsqz\" (UniqueName: \"kubernetes.io/projected/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d-kube-api-access-7xsqz\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:07.095116 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095079 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq"] Nov 03 23:06:07.095682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095668 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e1c20485-a969-4a59-a475-1dd186547b7b" containerName="revoke-keys" Nov 03 23:06:07.095744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095682 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c20485-a969-4a59-a475-1dd186547b7b" containerName="revoke-keys" Nov 03 23:06:07.095744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095704 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6edc41c7-1c97-4f3d-98b1-5182d9be168b" containerName="revoke-keys" Nov 03 23:06:07.095744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095711 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edc41c7-1c97-4f3d-98b1-5182d9be168b" containerName="revoke-keys" Nov 03 23:06:07.095744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095721 4903 cpu_manager.go:414] "containerMap: removing container" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerName="istio-proxy" Nov 03 23:06:07.095744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095725 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerName="istio-proxy" Nov 03 23:06:07.095744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095737 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" containerName="payload-pre-processing" Nov 03 23:06:07.095744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095741 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" containerName="payload-pre-processing" Nov 03 23:06:07.095992 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095834 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c20485-a969-4a59-a475-1dd186547b7b" containerName="revoke-keys" Nov 03 23:06:07.095992 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095845 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edc41c7-1c97-4f3d-98b1-5182d9be168b" containerName="revoke-keys" Nov 03 23:06:07.095992 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095851 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a9ee84-e898-4b08-8889-7f7c6c566ac0" containerName="istio-proxy" Nov 03 23:06:07.095992 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.095860 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" containerName="payload-pre-processing" Nov 03 23:06:07.099264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.099249 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.113503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.113480 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:07.113627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.113591 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:06:07.113681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.113652 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:07.114205 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.114185 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:07.118871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.118846 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq"] Nov 03 23:06:07.274933 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.274889 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.275121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.275004 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.275121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.275071 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttsv6\" (UniqueName: \"kubernetes.io/projected/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-kube-api-access-ttsv6\") pod \"maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.376768 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.376733 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.377034 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.376801 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.377034 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.376849 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttsv6\" (UniqueName: \"kubernetes.io/projected/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-kube-api-access-ttsv6\") pod \"maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.377639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.377607 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.379269 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.379248 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.398198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.398175 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttsv6\" (UniqueName: \"kubernetes.io/projected/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-kube-api-access-ttsv6\") pod \"maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.413072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.413046 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:07.533130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.533051 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" Nov 03 23:06:07.533130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.533070 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r" event={"ID":"b8744347-6c12-42c5-b7b3-9e5f1ebabd0d","Type":"ContainerDied","Data":"f2b6f9b18a22a567094cafb75dddec35f4539434cd8f7dfb4a251ec20146b08a"} Nov 03 23:06:07.533657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.533146 4903 scope.go:122] "RemoveContainer" containerID="d597d25a664cf0d3f1ad9bfe1d49f12080eae47ccf1f2e7dcdcc0ee9aa2b37ca" Nov 03 23:06:07.565803 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.565777 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt"] Nov 03 23:06:07.571095 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.571076 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.583516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.583493 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r"] Nov 03 23:06:07.588439 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.588402 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt"] Nov 03 23:06:07.591259 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.591238 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-2a2eca-66ff5f6f5f-8cw8r"] Nov 03 23:06:07.680649 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.680611 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkjh\" (UniqueName: \"kubernetes.io/projected/224eb29b-51c4-4ce6-a564-30ce813df0d2-kube-api-access-kfkjh\") pod \"maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.680819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.680676 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.680819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.680788 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.722035 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.722004 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq"] Nov 03 23:06:07.723932 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:07.723909 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bedbab_6ddd_4a91_bd60_38d799fa8e16.slice/crio-912bb701cf2210bd681ef287a5425d5bf0ad26ee4436daee585dec34819516f9 WatchSource:0}: Error finding container 912bb701cf2210bd681ef287a5425d5bf0ad26ee4436daee585dec34819516f9: Status 404 returned error can't find the container with id 912bb701cf2210bd681ef287a5425d5bf0ad26ee4436daee585dec34819516f9 Nov 03 23:06:07.736074 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.736048 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8744347-6c12-42c5-b7b3-9e5f1ebabd0d" path="/var/lib/kubelet/pods/b8744347-6c12-42c5-b7b3-9e5f1ebabd0d/volumes" Nov 03 23:06:07.781487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.781458 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkjh\" (UniqueName: \"kubernetes.io/projected/224eb29b-51c4-4ce6-a564-30ce813df0d2-kube-api-access-kfkjh\") pod \"maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.781645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.781521 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.781645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.781604 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.782237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.782217 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.783779 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.783732 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.803157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.803128 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkjh\" (UniqueName: \"kubernetes.io/projected/224eb29b-51c4-4ce6-a564-30ce813df0d2-kube-api-access-kfkjh\") pod \"maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:07.884676 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:07.884641 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:08.187606 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:08.187581 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt"] Nov 03 23:06:08.189545 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:08.189509 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224eb29b_51c4_4ce6_a564_30ce813df0d2.slice/crio-a617ca60617ed3b92f5b54d929034d0147d8001902dbd92b97ce3b789c7eb955 WatchSource:0}: Error finding container a617ca60617ed3b92f5b54d929034d0147d8001902dbd92b97ce3b789c7eb955: Status 404 returned error can't find the container with id a617ca60617ed3b92f5b54d929034d0147d8001902dbd92b97ce3b789c7eb955 Nov 03 23:06:08.542851 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:08.542809 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" event={"ID":"e5bedbab-6ddd-4a91-bd60-38d799fa8e16","Type":"ContainerStarted","Data":"963f27dbbf3fa1c91e8ad0f5b54b93b111611fc5da2fbbc6d387528d41727a5a"} Nov 03 23:06:08.543363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:08.542877 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" event={"ID":"e5bedbab-6ddd-4a91-bd60-38d799fa8e16","Type":"ContainerStarted","Data":"912bb701cf2210bd681ef287a5425d5bf0ad26ee4436daee585dec34819516f9"} Nov 03 23:06:08.544792 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:08.544761 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" event={"ID":"224eb29b-51c4-4ce6-a564-30ce813df0d2","Type":"ContainerStarted","Data":"9836b349d7265c6632c041286c57d7099ee1bb4790b8e0605b1c800def897ec5"} Nov 03 23:06:08.544912 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:08.544812 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" event={"ID":"224eb29b-51c4-4ce6-a564-30ce813df0d2","Type":"ContainerStarted","Data":"a617ca60617ed3b92f5b54d929034d0147d8001902dbd92b97ce3b789c7eb955"} Nov 03 23:06:08.581136 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:08.581086 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" podStartSLOduration=1.581070258 podStartE2EDuration="1.581070258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:08.578592689 +0000 UTC m=+3194.982533664" watchObservedRunningTime="2026-11-03 23:06:08.581070258 +0000 UTC m=+3194.985011233" Nov 03 23:06:08.615570 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:08.615446 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" podStartSLOduration=1.615407802 podStartE2EDuration="1.615407802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:08.614123084 +0000 UTC m=+3195.018064053" watchObservedRunningTime="2026-11-03 23:06:08.615407802 +0000 UTC m=+3195.019348773" Nov 03 23:06:09.554716 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:09.554681 4903 generic.go:356] "Generic (PLEG): container finished" podID="e5bedbab-6ddd-4a91-bd60-38d799fa8e16" containerID="963f27dbbf3fa1c91e8ad0f5b54b93b111611fc5da2fbbc6d387528d41727a5a" exitCode=0 Nov 03 23:06:09.555148 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:09.554765 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" event={"ID":"e5bedbab-6ddd-4a91-bd60-38d799fa8e16","Type":"ContainerDied","Data":"963f27dbbf3fa1c91e8ad0f5b54b93b111611fc5da2fbbc6d387528d41727a5a"} Nov 03 23:06:09.556492 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:09.556466 4903 generic.go:356] "Generic (PLEG): container finished" podID="224eb29b-51c4-4ce6-a564-30ce813df0d2" containerID="9836b349d7265c6632c041286c57d7099ee1bb4790b8e0605b1c800def897ec5" exitCode=0 Nov 03 23:06:09.556622 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:09.556547 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" event={"ID":"224eb29b-51c4-4ce6-a564-30ce813df0d2","Type":"ContainerDied","Data":"9836b349d7265c6632c041286c57d7099ee1bb4790b8e0605b1c800def897ec5"} Nov 03 23:06:10.745323 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.734722 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:10.790032 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.790000 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:10.820971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.820901 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-openshift-ca\") pod \"224eb29b-51c4-4ce6-a564-30ce813df0d2\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " Nov 03 23:06:10.821147 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.820987 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-openshift-ca\") pod \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " Nov 03 23:06:10.821147 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.821028 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-kube-api-access-ttsv6\" (UniqueName: \"kubernetes.io/projected/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-kube-api-access-ttsv6\") pod \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " Nov 03 23:06:10.821147 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.821086 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-xks-ca\") pod \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\" (UID: \"e5bedbab-6ddd-4a91-bd60-38d799fa8e16\") " Nov 03 23:06:10.821147 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.821116 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224eb29b-51c4-4ce6-a564-30ce813df0d2-kube-api-access-kfkjh\" (UniqueName: \"kubernetes.io/projected/224eb29b-51c4-4ce6-a564-30ce813df0d2-kube-api-access-kfkjh\") pod \"224eb29b-51c4-4ce6-a564-30ce813df0d2\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " Nov 03 23:06:10.821409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.821174 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-xks-ca\") pod \"224eb29b-51c4-4ce6-a564-30ce813df0d2\" (UID: \"224eb29b-51c4-4ce6-a564-30ce813df0d2\") " Nov 03 23:06:10.821409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.821349 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-openshift-ca" pod "224eb29b-51c4-4ce6-a564-30ce813df0d2" (UID: "224eb29b-51c4-4ce6-a564-30ce813df0d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:10.821507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.821425 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-openshift-ca" pod "e5bedbab-6ddd-4a91-bd60-38d799fa8e16" (UID: "e5bedbab-6ddd-4a91-bd60-38d799fa8e16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:10.821799 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.821775 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:10.821927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.821801 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:10.823525 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.823496 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-xks-ca" pod "224eb29b-51c4-4ce6-a564-30ce813df0d2" (UID: "224eb29b-51c4-4ce6-a564-30ce813df0d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:10.823700 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.823678 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-xks-ca" pod "e5bedbab-6ddd-4a91-bd60-38d799fa8e16" (UID: "e5bedbab-6ddd-4a91-bd60-38d799fa8e16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:10.823844 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.823828 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-kube-api-access-ttsv6" pod "e5bedbab-6ddd-4a91-bd60-38d799fa8e16" (UID: "e5bedbab-6ddd-4a91-bd60-38d799fa8e16"). InnerVolumeSpecName "kube-api-access-ttsv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:10.823981 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.823965 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224eb29b-51c4-4ce6-a564-30ce813df0d2-kube-api-access-kfkjh" pod "224eb29b-51c4-4ce6-a564-30ce813df0d2" (UID: "224eb29b-51c4-4ce6-a564-30ce813df0d2"). InnerVolumeSpecName "kube-api-access-kfkjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:10.922632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.922596 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttsv6\" (UniqueName: \"kubernetes.io/projected/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-kube-api-access-ttsv6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:10.922632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.922626 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5bedbab-6ddd-4a91-bd60-38d799fa8e16-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:10.922632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.922635 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfkjh\" (UniqueName: \"kubernetes.io/projected/224eb29b-51c4-4ce6-a564-30ce813df0d2-kube-api-access-kfkjh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:10.922890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:10.922644 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/224eb29b-51c4-4ce6-a564-30ce813df0d2-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:11.154230 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.154183 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w"] Nov 03 23:06:11.154844 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.154823 4903 cpu_manager.go:414] "containerMap: removing container" podUID="224eb29b-51c4-4ce6-a564-30ce813df0d2" containerName="revoke-keys" Nov 03 23:06:11.154844 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.154841 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224eb29b-51c4-4ce6-a564-30ce813df0d2" containerName="revoke-keys" Nov 03 23:06:11.154959 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.154864 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e5bedbab-6ddd-4a91-bd60-38d799fa8e16" containerName="revoke-keys" Nov 03 23:06:11.154959 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.154869 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bedbab-6ddd-4a91-bd60-38d799fa8e16" containerName="revoke-keys" Nov 03 23:06:11.155026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.154960 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="224eb29b-51c4-4ce6-a564-30ce813df0d2" containerName="revoke-keys" Nov 03 23:06:11.155026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.154977 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bedbab-6ddd-4a91-bd60-38d799fa8e16" containerName="revoke-keys" Nov 03 23:06:11.159751 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.159731 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.179532 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.179495 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w"] Nov 03 23:06:11.225491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.225443 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzg4\" (UniqueName: \"kubernetes.io/projected/6e0a0428-aefd-41d0-abb1-0b211b671d92-kube-api-access-cxzg4\") pod \"maas-api-revoke-sub-bc728923-qqh6w\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.225670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.225552 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc728923-qqh6w\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.225670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.225632 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc728923-qqh6w\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.326624 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.326587 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc728923-qqh6w\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.326859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.326749 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzg4\" (UniqueName: \"kubernetes.io/projected/6e0a0428-aefd-41d0-abb1-0b211b671d92-kube-api-access-cxzg4\") pod \"maas-api-revoke-sub-bc728923-qqh6w\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.326859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.326789 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc728923-qqh6w\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.327615 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.327550 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc728923-qqh6w\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.329880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.329852 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc728923-qqh6w\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.349033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.349003 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzg4\" (UniqueName: \"kubernetes.io/projected/6e0a0428-aefd-41d0-abb1-0b211b671d92-kube-api-access-cxzg4\") pod \"maas-api-revoke-sub-bc728923-qqh6w\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.477281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.477194 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:11.579704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.579672 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" Nov 03 23:06:11.579889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.579703 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq" event={"ID":"e5bedbab-6ddd-4a91-bd60-38d799fa8e16","Type":"ContainerDied","Data":"912bb701cf2210bd681ef287a5425d5bf0ad26ee4436daee585dec34819516f9"} Nov 03 23:06:11.579889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.579756 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="912bb701cf2210bd681ef287a5425d5bf0ad26ee4436daee585dec34819516f9" Nov 03 23:06:11.581363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.581335 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" event={"ID":"224eb29b-51c4-4ce6-a564-30ce813df0d2","Type":"ContainerDied","Data":"a617ca60617ed3b92f5b54d929034d0147d8001902dbd92b97ce3b789c7eb955"} Nov 03 23:06:11.581482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.581365 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt" Nov 03 23:06:11.581482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.581383 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a617ca60617ed3b92f5b54d929034d0147d8001902dbd92b97ce3b789c7eb955" Nov 03 23:06:11.612378 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.612342 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" podUID="2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" containerName="payload-processing" containerID="cri-o://f8447395ad8bc12c7623e00e428ea7e9da1253f159c963350f981e757fc741bb" gracePeriod=25 Nov 03 23:06:11.635229 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.635195 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n"] Nov 03 23:06:11.639339 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.639271 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" podUID="84afa961-a34b-44a0-9e02-68a6dc7a04dc" containerName="payload-pre-processing" containerID="cri-o://1d0b761642afb5ee2225bc6bf28f11373c431eef793bee72bcf5965c72fed8f3" gracePeriod=25 Nov 03 23:06:11.641737 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.641713 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8"] Nov 03 23:06:11.656901 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.656873 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4"] Nov 03 23:06:11.783684 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:11.783650 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0a0428_aefd_41d0_abb1_0b211b671d92.slice/crio-84edca7d9e94d21112353b167148fb21c9bb44175f43be8d799a83b7529eb3dc WatchSource:0}: Error finding container 84edca7d9e94d21112353b167148fb21c9bb44175f43be8d799a83b7529eb3dc: Status 404 returned error can't find the container with id 84edca7d9e94d21112353b167148fb21c9bb44175f43be8d799a83b7529eb3dc Nov 03 23:06:11.784050 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:11.783977 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w"] Nov 03 23:06:12.340763 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:12.340725 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g"] Nov 03 23:06:12.590103 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:12.590063 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" event={"ID":"6e0a0428-aefd-41d0-abb1-0b211b671d92","Type":"ContainerStarted","Data":"0a68921f768c5ae999603c7784d59b03315628b7bb51cec0d22b49632e87b297"} Nov 03 23:06:12.590272 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:12.590154 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" event={"ID":"6e0a0428-aefd-41d0-abb1-0b211b671d92","Type":"ContainerStarted","Data":"84edca7d9e94d21112353b167148fb21c9bb44175f43be8d799a83b7529eb3dc"} Nov 03 23:06:12.631097 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:12.631043 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" podStartSLOduration=1.6310290520000001 podStartE2EDuration="1.631029052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:12.628326558 +0000 UTC m=+3199.032267531" watchObservedRunningTime="2026-11-03 23:06:12.631029052 +0000 UTC m=+3199.034970026" Nov 03 23:06:12.692031 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:12.691860 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n"] Nov 03 23:06:12.695328 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:12.695301 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-26f8ba-nt24n"] Nov 03 23:06:13.599578 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:13.599539 4903 generic.go:356] "Generic (PLEG): container finished" podID="6e0a0428-aefd-41d0-abb1-0b211b671d92" containerID="0a68921f768c5ae999603c7784d59b03315628b7bb51cec0d22b49632e87b297" exitCode=0 Nov 03 23:06:13.600002 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:13.599623 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" event={"ID":"6e0a0428-aefd-41d0-abb1-0b211b671d92","Type":"ContainerDied","Data":"0a68921f768c5ae999603c7784d59b03315628b7bb51cec0d22b49632e87b297"} Nov 03 23:06:13.735967 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:13.735932 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa68a0b-cbd2-4569-b50f-7699a0aa45f1" path="/var/lib/kubelet/pods/0fa68a0b-cbd2-4569-b50f-7699a0aa45f1/volumes" Nov 03 23:06:14.797111 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.797083 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:14.870332 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.870208 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-xks-ca\") pod \"6e0a0428-aefd-41d0-abb1-0b211b671d92\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " Nov 03 23:06:14.870532 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.870423 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-openshift-ca\") pod \"6e0a0428-aefd-41d0-abb1-0b211b671d92\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " Nov 03 23:06:14.870613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.870531 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0a0428-aefd-41d0-abb1-0b211b671d92-kube-api-access-cxzg4\" (UniqueName: \"kubernetes.io/projected/6e0a0428-aefd-41d0-abb1-0b211b671d92-kube-api-access-cxzg4\") pod \"6e0a0428-aefd-41d0-abb1-0b211b671d92\" (UID: \"6e0a0428-aefd-41d0-abb1-0b211b671d92\") " Nov 03 23:06:14.870840 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.870809 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-openshift-ca" pod "6e0a0428-aefd-41d0-abb1-0b211b671d92" (UID: "6e0a0428-aefd-41d0-abb1-0b211b671d92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:14.871089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.871068 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:14.872831 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.872798 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0a0428-aefd-41d0-abb1-0b211b671d92-kube-api-access-cxzg4" pod "6e0a0428-aefd-41d0-abb1-0b211b671d92" (UID: "6e0a0428-aefd-41d0-abb1-0b211b671d92"). InnerVolumeSpecName "kube-api-access-cxzg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:14.874486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.874446 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-xks-ca" pod "6e0a0428-aefd-41d0-abb1-0b211b671d92" (UID: "6e0a0428-aefd-41d0-abb1-0b211b671d92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:14.972447 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.972401 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e0a0428-aefd-41d0-abb1-0b211b671d92-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:14.972447 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:14.972447 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzg4\" (UniqueName: \"kubernetes.io/projected/6e0a0428-aefd-41d0-abb1-0b211b671d92-kube-api-access-cxzg4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:15.298610 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:15.298562 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" podUID="84afa961-a34b-44a0-9e02-68a6dc7a04dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:15.620187 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:15.620106 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" Nov 03 23:06:15.620187 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:15.620140 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w" event={"ID":"6e0a0428-aefd-41d0-abb1-0b211b671d92","Type":"ContainerDied","Data":"84edca7d9e94d21112353b167148fb21c9bb44175f43be8d799a83b7529eb3dc"} Nov 03 23:06:15.620427 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:15.620189 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84edca7d9e94d21112353b167148fb21c9bb44175f43be8d799a83b7529eb3dc" Nov 03 23:06:16.315488 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:16.315453 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" podUID="2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:17.506181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:17.506149 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp"] Nov 03 23:06:17.541058 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:17.541026 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw"] Nov 03 23:06:18.163855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:18.163820 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq"] Nov 03 23:06:18.168358 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:18.168329 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0847aa-zb8nq"] Nov 03 23:06:18.216958 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:18.216925 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv"] Nov 03 23:06:18.316966 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:18.316936 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf"] Nov 03 23:06:18.387422 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:18.387377 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:06:19.725191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.725160 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9"] Nov 03 23:06:19.725964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.725946 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6e0a0428-aefd-41d0-abb1-0b211b671d92" containerName="revoke-keys" Nov 03 23:06:19.726029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.725966 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0a0428-aefd-41d0-abb1-0b211b671d92" containerName="revoke-keys" Nov 03 23:06:19.726067 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.726063 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0a0428-aefd-41d0-abb1-0b211b671d92" containerName="revoke-keys" Nov 03 23:06:19.729557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.729534 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:19.740035 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.740005 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c20485-a969-4a59-a475-1dd186547b7b" path="/var/lib/kubelet/pods/e1c20485-a969-4a59-a475-1dd186547b7b/volumes" Nov 03 23:06:19.761764 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.761738 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:19.761915 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.761778 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:19.762193 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.762178 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:19.763565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.763544 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:06:19.777739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.777706 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9"] Nov 03 23:06:19.818201 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.818167 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst"] Nov 03 23:06:19.822431 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.822414 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:19.833596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.833571 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62e2c3cb-kxxl9\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:19.833754 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.833730 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-kube-api-access-hknm2\") pod \"maas-api-revoke-sub-62e2c3cb-kxxl9\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:19.833821 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.833783 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62e2c3cb-kxxl9\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:19.841926 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.841902 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst"] Nov 03 23:06:19.934871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.934839 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zls27\" (UniqueName: \"kubernetes.io/projected/08eded23-13b5-438a-b832-30b199e9984c-kube-api-access-zls27\") pod \"maas-api-revoke-sub-6cd63b87-dmxst\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:19.935028 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.934967 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eded23-13b5-438a-b832-30b199e9984c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cd63b87-dmxst\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:19.935095 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.935043 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62e2c3cb-kxxl9\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:19.935095 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.935081 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eded23-13b5-438a-b832-30b199e9984c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cd63b87-dmxst\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:19.935164 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.935124 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-kube-api-access-hknm2\") pod \"maas-api-revoke-sub-62e2c3cb-kxxl9\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:19.935164 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.935147 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62e2c3cb-kxxl9\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:19.935815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.935793 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62e2c3cb-kxxl9\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:19.937484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.937453 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62e2c3cb-kxxl9\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:19.958722 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:19.958693 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-kube-api-access-hknm2\") pod \"maas-api-revoke-sub-62e2c3cb-kxxl9\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:20.036743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.036661 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eded23-13b5-438a-b832-30b199e9984c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cd63b87-dmxst\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:20.036929 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.036777 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zls27\" (UniqueName: \"kubernetes.io/projected/08eded23-13b5-438a-b832-30b199e9984c-kube-api-access-zls27\") pod \"maas-api-revoke-sub-6cd63b87-dmxst\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:20.036929 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.036846 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eded23-13b5-438a-b832-30b199e9984c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cd63b87-dmxst\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:20.037556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.037531 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eded23-13b5-438a-b832-30b199e9984c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cd63b87-dmxst\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:20.039219 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.039196 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eded23-13b5-438a-b832-30b199e9984c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cd63b87-dmxst\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:20.046711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.046688 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:20.059859 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.059795 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zls27\" (UniqueName: \"kubernetes.io/projected/08eded23-13b5-438a-b832-30b199e9984c-kube-api-access-zls27\") pod \"maas-api-revoke-sub-6cd63b87-dmxst\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:20.136417 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.136380 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:20.356582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.356555 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9"] Nov 03 23:06:20.357800 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:20.357775 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58e8ec9_0464_4e28_9bf1_bb6d8af30ae5.slice/crio-bc15b200f7b082dfe38aa1b2b27e5aad347bbfb6699bb6a038ab36b1c9ca6e20 WatchSource:0}: Error finding container bc15b200f7b082dfe38aa1b2b27e5aad347bbfb6699bb6a038ab36b1c9ca6e20: Status 404 returned error can't find the container with id bc15b200f7b082dfe38aa1b2b27e5aad347bbfb6699bb6a038ab36b1c9ca6e20 Nov 03 23:06:20.452094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.452064 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst"] Nov 03 23:06:20.667766 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.667726 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l"] Nov 03 23:06:20.668023 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.667984 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" containerID="cri-o://04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07" gracePeriod=30 Nov 03 23:06:20.670378 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.670274 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" event={"ID":"08eded23-13b5-438a-b832-30b199e9984c","Type":"ContainerStarted","Data":"3b324ea45d3051a8b41d5f12f7fafeec44740891238bd19f9983a24ab666c669"} Nov 03 23:06:20.672101 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:20.672049 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" event={"ID":"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5","Type":"ContainerStarted","Data":"bc15b200f7b082dfe38aa1b2b27e5aad347bbfb6699bb6a038ab36b1c9ca6e20"} Nov 03 23:06:21.370819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.370792 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt"] Nov 03 23:06:21.375085 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.375065 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.392519 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.392493 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt"] Nov 03 23:06:21.465503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.465474 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stfkz\" (UniqueName: \"kubernetes.io/projected/6567522c-adb2-4fe7-847b-8a330647ba9c-kube-api-access-stfkz\") pod \"maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.465620 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.465525 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.465709 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.465696 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.567361 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.567324 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.567576 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.567467 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stfkz\" (UniqueName: \"kubernetes.io/projected/6567522c-adb2-4fe7-847b-8a330647ba9c-kube-api-access-stfkz\") pod \"maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.567576 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.567514 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.568153 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.568129 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.570379 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.570339 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.589100 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.589066 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stfkz\" (UniqueName: \"kubernetes.io/projected/6567522c-adb2-4fe7-847b-8a330647ba9c-kube-api-access-stfkz\") pod \"maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:21.680746 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.680716 4903 generic.go:356] "Generic (PLEG): container finished" podID="08eded23-13b5-438a-b832-30b199e9984c" containerID="d1ca1478b4323b5f4a7d355e87a7720f46e15f1e2ea009713e71a3095c0b695b" exitCode=0 Nov 03 23:06:21.680933 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.680798 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" event={"ID":"08eded23-13b5-438a-b832-30b199e9984c","Type":"ContainerDied","Data":"d1ca1478b4323b5f4a7d355e87a7720f46e15f1e2ea009713e71a3095c0b695b"} Nov 03 23:06:21.682570 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.682545 4903 generic.go:356] "Generic (PLEG): container finished" podID="c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5" containerID="f2e7d9cf32ae46b07b4c66173d38040c39a81a85c42f7d4c883056a70e757cc0" exitCode=0 Nov 03 23:06:21.682684 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.682583 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" event={"ID":"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5","Type":"ContainerDied","Data":"f2e7d9cf32ae46b07b4c66173d38040c39a81a85c42f7d4c883056a70e757cc0"} Nov 03 23:06:21.719516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:21.719486 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:22.033084 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.033051 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt"] Nov 03 23:06:22.034520 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:22.034489 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6567522c_adb2_4fe7_847b_8a330647ba9c.slice/crio-f272ad3819a23e5a80ffddf8272a9530eaefe1603ddd8d51e331d803e5bb684f WatchSource:0}: Error finding container f272ad3819a23e5a80ffddf8272a9530eaefe1603ddd8d51e331d803e5bb684f: Status 404 returned error can't find the container with id f272ad3819a23e5a80ffddf8272a9530eaefe1603ddd8d51e331d803e5bb684f Nov 03 23:06:22.167198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.167163 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75"] Nov 03 23:06:22.506697 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.506651 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" podUID="f41e7d90-969b-4616-be62-c357c98e70a0" containerName="payload-processing" containerID="cri-o://6512f0cff1d703cca3fad666471bddec4f64daa8d001438a211196959000f37a" gracePeriod=25 Nov 03 23:06:22.541654 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.541591 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" podUID="c4a2f30c-1a18-40be-b501-0571787facdc" containerName="payload-pre-processing" containerID="cri-o://fdd152ad35bfd985058efb2816fe9ac1477d3ea9ccca1feb901001a864df37dc" gracePeriod=25 Nov 03 23:06:22.694115 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.694074 4903 generic.go:356] "Generic (PLEG): container finished" podID="f41e7d90-969b-4616-be62-c357c98e70a0" containerID="6512f0cff1d703cca3fad666471bddec4f64daa8d001438a211196959000f37a" exitCode=0 Nov 03 23:06:22.694315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.694150 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" event={"ID":"f41e7d90-969b-4616-be62-c357c98e70a0","Type":"ContainerDied","Data":"6512f0cff1d703cca3fad666471bddec4f64daa8d001438a211196959000f37a"} Nov 03 23:06:22.696438 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.696278 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" event={"ID":"6567522c-adb2-4fe7-847b-8a330647ba9c","Type":"ContainerStarted","Data":"98b9e306ef271b124efdd47a2d2c93567823edf3e333cbdef9704223e6c4e3c6"} Nov 03 23:06:22.696438 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.696354 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" event={"ID":"6567522c-adb2-4fe7-847b-8a330647ba9c","Type":"ContainerStarted","Data":"f272ad3819a23e5a80ffddf8272a9530eaefe1603ddd8d51e331d803e5bb684f"} Nov 03 23:06:22.698211 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.698185 4903 generic.go:356] "Generic (PLEG): container finished" podID="c4a2f30c-1a18-40be-b501-0571787facdc" containerID="fdd152ad35bfd985058efb2816fe9ac1477d3ea9ccca1feb901001a864df37dc" exitCode=0 Nov 03 23:06:22.698358 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.698257 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" event={"ID":"c4a2f30c-1a18-40be-b501-0571787facdc","Type":"ContainerDied","Data":"fdd152ad35bfd985058efb2816fe9ac1477d3ea9ccca1feb901001a864df37dc"} Nov 03 23:06:22.756552 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.753846 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" podStartSLOduration=1.7538280130000001 podStartE2EDuration="1.753828013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:22.73836545 +0000 UTC m=+3209.142306426" watchObservedRunningTime="2026-11-03 23:06:22.753828013 +0000 UTC m=+3209.157768988" Nov 03 23:06:22.816313 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.816271 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 23:06:22.886645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.886612 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41e7d90-969b-4616-be62-c357c98e70a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41e7d90-969b-4616-be62-c357c98e70a0-plugins-config-volume\") pod \"f41e7d90-969b-4616-be62-c357c98e70a0\" (UID: \"f41e7d90-969b-4616-be62-c357c98e70a0\") " Nov 03 23:06:22.886825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.886804 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41e7d90-969b-4616-be62-c357c98e70a0-kube-api-access-hngf8\" (UniqueName: \"kubernetes.io/projected/f41e7d90-969b-4616-be62-c357c98e70a0-kube-api-access-hngf8\") pod \"f41e7d90-969b-4616-be62-c357c98e70a0\" (UID: \"f41e7d90-969b-4616-be62-c357c98e70a0\") " Nov 03 23:06:22.887072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.887045 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41e7d90-969b-4616-be62-c357c98e70a0-plugins-config-volume" pod "f41e7d90-969b-4616-be62-c357c98e70a0" (UID: "f41e7d90-969b-4616-be62-c357c98e70a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:22.887448 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.887424 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f41e7d90-969b-4616-be62-c357c98e70a0-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:22.889934 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.889765 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41e7d90-969b-4616-be62-c357c98e70a0-kube-api-access-hngf8" pod "f41e7d90-969b-4616-be62-c357c98e70a0" (UID: "f41e7d90-969b-4616-be62-c357c98e70a0"). InnerVolumeSpecName "kube-api-access-hngf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:22.988514 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:22.988483 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hngf8\" (UniqueName: \"kubernetes.io/projected/f41e7d90-969b-4616-be62-c357c98e70a0-kube-api-access-hngf8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.041466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.041435 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 23:06:23.046179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.046156 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:23.048793 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.048772 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:23.089239 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.089200 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a2f30c-1a18-40be-b501-0571787facdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a2f30c-1a18-40be-b501-0571787facdc-plugins-config-volume\") pod \"c4a2f30c-1a18-40be-b501-0571787facdc\" (UID: \"c4a2f30c-1a18-40be-b501-0571787facdc\") " Nov 03 23:06:23.089443 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.089401 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a2f30c-1a18-40be-b501-0571787facdc-kube-api-access-vfrh4\" (UniqueName: \"kubernetes.io/projected/c4a2f30c-1a18-40be-b501-0571787facdc-kube-api-access-vfrh4\") pod \"c4a2f30c-1a18-40be-b501-0571787facdc\" (UID: \"c4a2f30c-1a18-40be-b501-0571787facdc\") " Nov 03 23:06:23.089616 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.089592 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a2f30c-1a18-40be-b501-0571787facdc-plugins-config-volume" pod "c4a2f30c-1a18-40be-b501-0571787facdc" (UID: "c4a2f30c-1a18-40be-b501-0571787facdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:23.089847 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.089830 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4a2f30c-1a18-40be-b501-0571787facdc-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.091787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.091761 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a2f30c-1a18-40be-b501-0571787facdc-kube-api-access-vfrh4" pod "c4a2f30c-1a18-40be-b501-0571787facdc" (UID: "c4a2f30c-1a18-40be-b501-0571787facdc"). InnerVolumeSpecName "kube-api-access-vfrh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:23.190660 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.190626 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-openshift-ca\") pod \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " Nov 03 23:06:23.190830 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.190699 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08eded23-13b5-438a-b832-30b199e9984c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eded23-13b5-438a-b832-30b199e9984c-maas-api-xks-ca\") pod \"08eded23-13b5-438a-b832-30b199e9984c\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " Nov 03 23:06:23.190830 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.190738 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-xks-ca\") pod \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " Nov 03 23:06:23.190830 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.190779 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08eded23-13b5-438a-b832-30b199e9984c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eded23-13b5-438a-b832-30b199e9984c-maas-api-openshift-ca\") pod \"08eded23-13b5-438a-b832-30b199e9984c\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " Nov 03 23:06:23.191041 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.190843 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-kube-api-access-hknm2\") pod \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\" (UID: \"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5\") " Nov 03 23:06:23.191041 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.191015 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08eded23-13b5-438a-b832-30b199e9984c-kube-api-access-zls27\" (UniqueName: \"kubernetes.io/projected/08eded23-13b5-438a-b832-30b199e9984c-kube-api-access-zls27\") pod \"08eded23-13b5-438a-b832-30b199e9984c\" (UID: \"08eded23-13b5-438a-b832-30b199e9984c\") " Nov 03 23:06:23.191156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.191067 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-openshift-ca" pod "c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5" (UID: "c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:23.191626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.191578 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.191626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.191603 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfrh4\" (UniqueName: \"kubernetes.io/projected/c4a2f30c-1a18-40be-b501-0571787facdc-kube-api-access-vfrh4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.191839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.191805 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08eded23-13b5-438a-b832-30b199e9984c-maas-api-openshift-ca" pod "08eded23-13b5-438a-b832-30b199e9984c" (UID: "08eded23-13b5-438a-b832-30b199e9984c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:23.193391 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.193366 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-xks-ca" pod "c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5" (UID: "c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:23.193537 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.193518 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-kube-api-access-hknm2" pod "c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5" (UID: "c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5"). InnerVolumeSpecName "kube-api-access-hknm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:23.194045 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.194028 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08eded23-13b5-438a-b832-30b199e9984c-maas-api-xks-ca" pod "08eded23-13b5-438a-b832-30b199e9984c" (UID: "08eded23-13b5-438a-b832-30b199e9984c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:23.194950 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.194926 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eded23-13b5-438a-b832-30b199e9984c-kube-api-access-zls27" pod "08eded23-13b5-438a-b832-30b199e9984c" (UID: "08eded23-13b5-438a-b832-30b199e9984c"). InnerVolumeSpecName "kube-api-access-zls27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:23.217970 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.217930 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" podUID="58aedb13-1685-4dfb-a138-af48cd8e52a9" containerName="payload-processing" containerID="cri-o://6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9" gracePeriod=25 Nov 03 23:06:23.292909 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.292876 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08eded23-13b5-438a-b832-30b199e9984c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.292909 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.292904 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknm2\" (UniqueName: \"kubernetes.io/projected/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-kube-api-access-hknm2\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.292909 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.292914 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zls27\" (UniqueName: \"kubernetes.io/projected/08eded23-13b5-438a-b832-30b199e9984c-kube-api-access-zls27\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.293115 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.292926 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08eded23-13b5-438a-b832-30b199e9984c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.293115 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.292935 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.318064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.318029 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" podUID="2a594be9-4d23-4125-9a34-76e2febb1936" containerName="payload-pre-processing" containerID="cri-o://00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512" gracePeriod=25 Nov 03 23:06:23.476288 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.476259 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 23:06:23.598324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.596771 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aedb13-1685-4dfb-a138-af48cd8e52a9-kube-api-access-5rvkx\" (UniqueName: \"kubernetes.io/projected/58aedb13-1685-4dfb-a138-af48cd8e52a9-kube-api-access-5rvkx\") pod \"58aedb13-1685-4dfb-a138-af48cd8e52a9\" (UID: \"58aedb13-1685-4dfb-a138-af48cd8e52a9\") " Nov 03 23:06:23.598324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.596920 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58aedb13-1685-4dfb-a138-af48cd8e52a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aedb13-1685-4dfb-a138-af48cd8e52a9-plugins-config-volume\") pod \"58aedb13-1685-4dfb-a138-af48cd8e52a9\" (UID: \"58aedb13-1685-4dfb-a138-af48cd8e52a9\") " Nov 03 23:06:23.603241 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.602041 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aedb13-1685-4dfb-a138-af48cd8e52a9-plugins-config-volume" pod "58aedb13-1685-4dfb-a138-af48cd8e52a9" (UID: "58aedb13-1685-4dfb-a138-af48cd8e52a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:23.606135 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.606048 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aedb13-1685-4dfb-a138-af48cd8e52a9-kube-api-access-5rvkx" pod "58aedb13-1685-4dfb-a138-af48cd8e52a9" (UID: "58aedb13-1685-4dfb-a138-af48cd8e52a9"). InnerVolumeSpecName "kube-api-access-5rvkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:23.616741 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.616719 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 23:06:23.698356 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.698319 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a594be9-4d23-4125-9a34-76e2febb1936-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a594be9-4d23-4125-9a34-76e2febb1936-plugins-config-volume\") pod \"2a594be9-4d23-4125-9a34-76e2febb1936\" (UID: \"2a594be9-4d23-4125-9a34-76e2febb1936\") " Nov 03 23:06:23.698532 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.698446 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a594be9-4d23-4125-9a34-76e2febb1936-kube-api-access-5cpxc\" (UniqueName: \"kubernetes.io/projected/2a594be9-4d23-4125-9a34-76e2febb1936-kube-api-access-5cpxc\") pod \"2a594be9-4d23-4125-9a34-76e2febb1936\" (UID: \"2a594be9-4d23-4125-9a34-76e2febb1936\") " Nov 03 23:06:23.698741 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.698704 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a594be9-4d23-4125-9a34-76e2febb1936-plugins-config-volume" pod "2a594be9-4d23-4125-9a34-76e2febb1936" (UID: "2a594be9-4d23-4125-9a34-76e2febb1936"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:23.698911 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.698893 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rvkx\" (UniqueName: \"kubernetes.io/projected/58aedb13-1685-4dfb-a138-af48cd8e52a9-kube-api-access-5rvkx\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.698970 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.698916 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a594be9-4d23-4125-9a34-76e2febb1936-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.698970 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.698930 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aedb13-1685-4dfb-a138-af48cd8e52a9-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.700568 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.700542 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a594be9-4d23-4125-9a34-76e2febb1936-kube-api-access-5cpxc" pod "2a594be9-4d23-4125-9a34-76e2febb1936" (UID: "2a594be9-4d23-4125-9a34-76e2febb1936"). InnerVolumeSpecName "kube-api-access-5cpxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:23.707699 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.707674 4903 generic.go:356] "Generic (PLEG): container finished" podID="58aedb13-1685-4dfb-a138-af48cd8e52a9" containerID="6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9" exitCode=0 Nov 03 23:06:23.707811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.707714 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" Nov 03 23:06:23.707811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.707754 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" event={"ID":"58aedb13-1685-4dfb-a138-af48cd8e52a9","Type":"ContainerDied","Data":"6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9"} Nov 03 23:06:23.707911 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.707828 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv" event={"ID":"58aedb13-1685-4dfb-a138-af48cd8e52a9","Type":"ContainerDied","Data":"078cf241e8474c3cf8ad9a85c525edca44ff1b8800691b3cef7108336aac63c7"} Nov 03 23:06:23.707911 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.707851 4903 scope.go:122] "RemoveContainer" containerID="6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9" Nov 03 23:06:23.709352 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.709325 4903 generic.go:356] "Generic (PLEG): container finished" podID="2a594be9-4d23-4125-9a34-76e2febb1936" containerID="00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512" exitCode=0 Nov 03 23:06:23.709471 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.709379 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" Nov 03 23:06:23.709471 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.709409 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" event={"ID":"2a594be9-4d23-4125-9a34-76e2febb1936","Type":"ContainerDied","Data":"00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512"} Nov 03 23:06:23.709471 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.709453 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf" event={"ID":"2a594be9-4d23-4125-9a34-76e2febb1936","Type":"ContainerDied","Data":"f7d1fb476743531b3c9f962a263df5681c148ca2cde8ea6ef378f76b002b9f38"} Nov 03 23:06:23.711073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.711053 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" Nov 03 23:06:23.711191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.711056 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp" event={"ID":"f41e7d90-969b-4616-be62-c357c98e70a0","Type":"ContainerDied","Data":"87bea4839395af79c58c6a00413c6e8dcc01b8e0d190190e9f946adc87f4b2e1"} Nov 03 23:06:23.712819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.712797 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" event={"ID":"08eded23-13b5-438a-b832-30b199e9984c","Type":"ContainerDied","Data":"3b324ea45d3051a8b41d5f12f7fafeec44740891238bd19f9983a24ab666c669"} Nov 03 23:06:23.712979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.712963 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b324ea45d3051a8b41d5f12f7fafeec44740891238bd19f9983a24ab666c669" Nov 03 23:06:23.713095 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.712806 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst" Nov 03 23:06:23.716864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.716718 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" Nov 03 23:06:23.716864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.716760 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9" event={"ID":"c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5","Type":"ContainerDied","Data":"bc15b200f7b082dfe38aa1b2b27e5aad347bbfb6699bb6a038ab36b1c9ca6e20"} Nov 03 23:06:23.716864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.716799 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc15b200f7b082dfe38aa1b2b27e5aad347bbfb6699bb6a038ab36b1c9ca6e20" Nov 03 23:06:23.719314 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.719098 4903 generic.go:356] "Generic (PLEG): container finished" podID="6567522c-adb2-4fe7-847b-8a330647ba9c" containerID="98b9e306ef271b124efdd47a2d2c93567823edf3e333cbdef9704223e6c4e3c6" exitCode=0 Nov 03 23:06:23.719314 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.719169 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" event={"ID":"6567522c-adb2-4fe7-847b-8a330647ba9c","Type":"ContainerDied","Data":"98b9e306ef271b124efdd47a2d2c93567823edf3e333cbdef9704223e6c4e3c6"} Nov 03 23:06:23.723681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.722542 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" event={"ID":"c4a2f30c-1a18-40be-b501-0571787facdc","Type":"ContainerDied","Data":"7b99bc112bc457f2e076f47f974c270c022c959f5504ad095dc99651c410ea82"} Nov 03 23:06:23.723681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.722604 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw" Nov 03 23:06:23.724838 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.724818 4903 scope.go:122] "RemoveContainer" containerID="6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9" Nov 03 23:06:23.725486 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:23.725152 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9\": container with ID starting with 6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9 not found: ID does not exist" containerID="6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9" Nov 03 23:06:23.725486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.725190 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9"} err="failed to get container status \"6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9\": rpc error: code = NotFound desc = could not find container \"6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9\": container with ID starting with 6b6e1eda1ff23d7397b4ddbdd4fcdb8657dd4b6287fb42dd5c8a49a4aa9570f9 not found: ID does not exist" Nov 03 23:06:23.725486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.725213 4903 scope.go:122] "RemoveContainer" containerID="00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512" Nov 03 23:06:23.741763 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.741739 4903 scope.go:122] "RemoveContainer" containerID="00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512" Nov 03 23:06:23.742427 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:23.742399 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512\": container with ID starting with 00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512 not found: ID does not exist" containerID="00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512" Nov 03 23:06:23.742554 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.742435 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512"} err="failed to get container status \"00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512\": rpc error: code = NotFound desc = could not find container \"00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512\": container with ID starting with 00779e48679d18f76fa59a6ae9cde5bd6737c0ac66b8106c78f2e7a531d7b512 not found: ID does not exist" Nov 03 23:06:23.742554 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.742461 4903 scope.go:122] "RemoveContainer" containerID="6512f0cff1d703cca3fad666471bddec4f64daa8d001438a211196959000f37a" Nov 03 23:06:23.754098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.754080 4903 scope.go:122] "RemoveContainer" containerID="fdd152ad35bfd985058efb2816fe9ac1477d3ea9ccca1feb901001a864df37dc" Nov 03 23:06:23.800790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.800758 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpxc\" (UniqueName: \"kubernetes.io/projected/2a594be9-4d23-4125-9a34-76e2febb1936-kube-api-access-5cpxc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:23.842712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.842658 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf"] Nov 03 23:06:23.849016 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.848985 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-50d738-5b4664c9c9-zldwf"] Nov 03 23:06:23.878643 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.878612 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw"] Nov 03 23:06:23.882869 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.882844 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-434ce3-7b5c6dc5b7-pgcvw"] Nov 03 23:06:23.940222 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.940192 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv"] Nov 03 23:06:23.946215 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.946191 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-50d738-85555f75b4-p55sv"] Nov 03 23:06:23.976203 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.976116 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp"] Nov 03 23:06:23.979403 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:23.979373 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-434ce3-5797fc889b-dfrzp"] Nov 03 23:06:24.899861 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:24.899835 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:25.019787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.017274 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6567522c-adb2-4fe7-847b-8a330647ba9c-kube-api-access-stfkz\" (UniqueName: \"kubernetes.io/projected/6567522c-adb2-4fe7-847b-8a330647ba9c-kube-api-access-stfkz\") pod \"6567522c-adb2-4fe7-847b-8a330647ba9c\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " Nov 03 23:06:25.019787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.017494 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-openshift-ca\") pod \"6567522c-adb2-4fe7-847b-8a330647ba9c\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " Nov 03 23:06:25.019787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.017659 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-xks-ca\") pod \"6567522c-adb2-4fe7-847b-8a330647ba9c\" (UID: \"6567522c-adb2-4fe7-847b-8a330647ba9c\") " Nov 03 23:06:25.019787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.018065 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-openshift-ca" pod "6567522c-adb2-4fe7-847b-8a330647ba9c" (UID: "6567522c-adb2-4fe7-847b-8a330647ba9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:25.019787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.018686 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:25.020777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.020269 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6567522c-adb2-4fe7-847b-8a330647ba9c-kube-api-access-stfkz" pod "6567522c-adb2-4fe7-847b-8a330647ba9c" (UID: "6567522c-adb2-4fe7-847b-8a330647ba9c"). InnerVolumeSpecName "kube-api-access-stfkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:25.020777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.020392 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-xks-ca" pod "6567522c-adb2-4fe7-847b-8a330647ba9c" (UID: "6567522c-adb2-4fe7-847b-8a330647ba9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:25.119835 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.119792 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6567522c-adb2-4fe7-847b-8a330647ba9c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:25.119835 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.119832 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stfkz\" (UniqueName: \"kubernetes.io/projected/6567522c-adb2-4fe7-847b-8a330647ba9c-kube-api-access-stfkz\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:25.131258 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131227 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm"] Nov 03 23:06:25.131821 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131805 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5" containerName="revoke-keys" Nov 03 23:06:25.131864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131823 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5" containerName="revoke-keys" Nov 03 23:06:25.131864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131840 4903 cpu_manager.go:414] "containerMap: removing container" podUID="58aedb13-1685-4dfb-a138-af48cd8e52a9" containerName="payload-processing" Nov 03 23:06:25.131864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131845 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aedb13-1685-4dfb-a138-af48cd8e52a9" containerName="payload-processing" Nov 03 23:06:25.131864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131855 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c4a2f30c-1a18-40be-b501-0571787facdc" containerName="payload-pre-processing" Nov 03 23:06:25.131864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131860 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a2f30c-1a18-40be-b501-0571787facdc" containerName="payload-pre-processing" Nov 03 23:06:25.132020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131870 4903 cpu_manager.go:414] "containerMap: removing container" podUID="08eded23-13b5-438a-b832-30b199e9984c" containerName="revoke-keys" Nov 03 23:06:25.132020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131875 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08eded23-13b5-438a-b832-30b199e9984c" containerName="revoke-keys" Nov 03 23:06:25.132020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131892 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f41e7d90-969b-4616-be62-c357c98e70a0" containerName="payload-processing" Nov 03 23:06:25.132020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131897 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41e7d90-969b-4616-be62-c357c98e70a0" containerName="payload-processing" Nov 03 23:06:25.132020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131904 4903 cpu_manager.go:414] "containerMap: removing container" podUID="6567522c-adb2-4fe7-847b-8a330647ba9c" containerName="revoke-keys" Nov 03 23:06:25.132020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131908 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6567522c-adb2-4fe7-847b-8a330647ba9c" containerName="revoke-keys" Nov 03 23:06:25.132020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131931 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2a594be9-4d23-4125-9a34-76e2febb1936" containerName="payload-pre-processing" Nov 03 23:06:25.132020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.131937 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a594be9-4d23-4125-9a34-76e2febb1936" containerName="payload-pre-processing" Nov 03 23:06:25.132020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.132019 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5" containerName="revoke-keys" Nov 03 23:06:25.132336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.132027 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aedb13-1685-4dfb-a138-af48cd8e52a9" containerName="payload-processing" Nov 03 23:06:25.132336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.132034 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a2f30c-1a18-40be-b501-0571787facdc" containerName="payload-pre-processing" Nov 03 23:06:25.132336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.132041 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="08eded23-13b5-438a-b832-30b199e9984c" containerName="revoke-keys" Nov 03 23:06:25.132336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.132050 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41e7d90-969b-4616-be62-c357c98e70a0" containerName="payload-processing" Nov 03 23:06:25.132336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.132054 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="6567522c-adb2-4fe7-847b-8a330647ba9c" containerName="revoke-keys" Nov 03 23:06:25.132336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.132063 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a594be9-4d23-4125-9a34-76e2febb1936" containerName="payload-pre-processing" Nov 03 23:06:25.137523 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.137501 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.171484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.171449 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw"] Nov 03 23:06:25.176652 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.176564 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.177284 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.176813 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm"] Nov 03 23:06:25.207305 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.207259 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw"] Nov 03 23:06:25.220984 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.220949 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rvdh\" (UniqueName: \"kubernetes.io/projected/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-kube-api-access-2rvdh\") pod \"maas-api-revoke-sub-4986d175-7cgkm\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.221362 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.221240 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4986d175-7cgkm\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.221498 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.221378 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4986d175-7cgkm\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.298281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.298201 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" podUID="84afa961-a34b-44a0-9e02-68a6dc7a04dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:25.323352 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.323310 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mspvm\" (UniqueName: \"kubernetes.io/projected/4017b29f-6cd4-484e-a0be-e97243142e4f-kube-api-access-mspvm\") pod \"maas-api-revoke-sub-be1b8fc6-2grgw\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.323525 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.323446 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be1b8fc6-2grgw\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.323617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.323531 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4986d175-7cgkm\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.323681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.323659 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rvdh\" (UniqueName: \"kubernetes.io/projected/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-kube-api-access-2rvdh\") pod \"maas-api-revoke-sub-4986d175-7cgkm\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.323781 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.323764 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be1b8fc6-2grgw\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.323846 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.323818 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4986d175-7cgkm\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.324541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.324518 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4986d175-7cgkm\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.326980 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.326951 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4986d175-7cgkm\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.358206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.358173 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rvdh\" (UniqueName: \"kubernetes.io/projected/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-kube-api-access-2rvdh\") pod \"maas-api-revoke-sub-4986d175-7cgkm\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.425011 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.424972 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be1b8fc6-2grgw\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.425208 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.425192 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be1b8fc6-2grgw\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.425279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.425259 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mspvm\" (UniqueName: \"kubernetes.io/projected/4017b29f-6cd4-484e-a0be-e97243142e4f-kube-api-access-mspvm\") pod \"maas-api-revoke-sub-be1b8fc6-2grgw\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.425991 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.425963 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be1b8fc6-2grgw\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.427462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.427430 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be1b8fc6-2grgw\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.446241 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.446221 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mspvm\" (UniqueName: \"kubernetes.io/projected/4017b29f-6cd4-484e-a0be-e97243142e4f-kube-api-access-mspvm\") pod \"maas-api-revoke-sub-be1b8fc6-2grgw\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.454112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.454092 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:25.512713 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.511092 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:25.734251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.734220 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a594be9-4d23-4125-9a34-76e2febb1936" path="/var/lib/kubelet/pods/2a594be9-4d23-4125-9a34-76e2febb1936/volumes" Nov 03 23:06:25.734635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.734619 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aedb13-1685-4dfb-a138-af48cd8e52a9" path="/var/lib/kubelet/pods/58aedb13-1685-4dfb-a138-af48cd8e52a9/volumes" Nov 03 23:06:25.734987 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.734974 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a2f30c-1a18-40be-b501-0571787facdc" path="/var/lib/kubelet/pods/c4a2f30c-1a18-40be-b501-0571787facdc/volumes" Nov 03 23:06:25.735354 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.735342 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41e7d90-969b-4616-be62-c357c98e70a0" path="/var/lib/kubelet/pods/f41e7d90-969b-4616-be62-c357c98e70a0/volumes" Nov 03 23:06:25.748095 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.748071 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" Nov 03 23:06:25.748215 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.748159 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt" event={"ID":"6567522c-adb2-4fe7-847b-8a330647ba9c","Type":"ContainerDied","Data":"f272ad3819a23e5a80ffddf8272a9530eaefe1603ddd8d51e331d803e5bb684f"} Nov 03 23:06:25.748215 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.748200 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f272ad3819a23e5a80ffddf8272a9530eaefe1603ddd8d51e331d803e5bb684f" Nov 03 23:06:25.760192 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.760164 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm"] Nov 03 23:06:25.761240 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:25.761202 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6a9ff7_6e2a_420d_bc42_847c128a1b6c.slice/crio-6a11d7d79ac6b4716a8e216879537c918cfc96b66c31261fc69e229f2b52877b WatchSource:0}: Error finding container 6a11d7d79ac6b4716a8e216879537c918cfc96b66c31261fc69e229f2b52877b: Status 404 returned error can't find the container with id 6a11d7d79ac6b4716a8e216879537c918cfc96b66c31261fc69e229f2b52877b Nov 03 23:06:25.786848 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.786817 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z"] Nov 03 23:06:25.792842 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.792776 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:25.810659 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.810632 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z"] Nov 03 23:06:25.845373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.845069 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw"] Nov 03 23:06:25.937324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.937279 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:25.937705 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.937403 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:25.937705 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.937473 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5tfg\" (UniqueName: \"kubernetes.io/projected/aad41311-0910-4d7f-980f-9c4062aa48c1-kube-api-access-c5tfg\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:25.943013 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:25.942990 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4017b29f_6cd4_484e_a0be_e97243142e4f.slice/crio-294e7ff155ccf5c23abc5578def3f3aec6a22d22ab44caea173680bcfff291e3 WatchSource:0}: Error finding container 294e7ff155ccf5c23abc5578def3f3aec6a22d22ab44caea173680bcfff291e3: Status 404 returned error can't find the container with id 294e7ff155ccf5c23abc5578def3f3aec6a22d22ab44caea173680bcfff291e3 Nov 03 23:06:25.953982 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:25.953961 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 23:06:26.038798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.038756 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-socket\") pod \"2fb0da7d-26b1-4802-b507-b552705b9d30\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " Nov 03 23:06:26.038977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.038867 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-credential-socket\") pod \"2fb0da7d-26b1-4802-b507-b552705b9d30\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " Nov 03 23:06:26.039045 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.038978 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-podinfo\") pod \"2fb0da7d-26b1-4802-b507-b552705b9d30\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " Nov 03 23:06:26.039045 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039014 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-certs\") pod \"2fb0da7d-26b1-4802-b507-b552705b9d30\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " Nov 03 23:06:26.039180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039050 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-token\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-token\") pod \"2fb0da7d-26b1-4802-b507-b552705b9d30\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " Nov 03 23:06:26.039180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039082 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-socket" pod "2fb0da7d-26b1-4802-b507-b552705b9d30" (UID: "2fb0da7d-26b1-4802-b507-b552705b9d30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:26.039180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039110 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-envoy\") pod \"2fb0da7d-26b1-4802-b507-b552705b9d30\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " Nov 03 23:06:26.039180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039158 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb0da7d-26b1-4802-b507-b552705b9d30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb0da7d-26b1-4802-b507-b552705b9d30-istiod-ca-cert\") pod \"2fb0da7d-26b1-4802-b507-b552705b9d30\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " Nov 03 23:06:26.039424 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039246 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-data\") pod \"2fb0da7d-26b1-4802-b507-b552705b9d30\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " Nov 03 23:06:26.039424 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039261 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-credential-socket" pod "2fb0da7d-26b1-4802-b507-b552705b9d30" (UID: "2fb0da7d-26b1-4802-b507-b552705b9d30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:26.039424 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039331 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-kube-api-access-2n9v6\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-kube-api-access-2n9v6\") pod \"2fb0da7d-26b1-4802-b507-b552705b9d30\" (UID: \"2fb0da7d-26b1-4802-b507-b552705b9d30\") " Nov 03 23:06:26.039603 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039580 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:26.039681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039588 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-data" pod "2fb0da7d-26b1-4802-b507-b552705b9d30" (UID: "2fb0da7d-26b1-4802-b507-b552705b9d30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:26.039681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039666 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:26.039789 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039733 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5tfg\" (UniqueName: \"kubernetes.io/projected/aad41311-0910-4d7f-980f-9c4062aa48c1-kube-api-access-c5tfg\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:26.039789 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.039761 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb0da7d-26b1-4802-b507-b552705b9d30-istiod-ca-cert" pod "2fb0da7d-26b1-4802-b507-b552705b9d30" (UID: "2fb0da7d-26b1-4802-b507-b552705b9d30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:26.040040 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.040018 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:26.040122 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.040045 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:26.040122 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.040060 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fb0da7d-26b1-4802-b507-b552705b9d30-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:26.040122 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.040073 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:26.040712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.040354 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-certs" pod "2fb0da7d-26b1-4802-b507-b552705b9d30" (UID: "2fb0da7d-26b1-4802-b507-b552705b9d30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:26.040712 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.040612 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:26.042037 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.041995 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-envoy" pod "2fb0da7d-26b1-4802-b507-b552705b9d30" (UID: "2fb0da7d-26b1-4802-b507-b552705b9d30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:26.043366 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.042986 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-podinfo" pod "2fb0da7d-26b1-4802-b507-b552705b9d30" (UID: "2fb0da7d-26b1-4802-b507-b552705b9d30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:26.043366 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.043060 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:26.043753 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.043727 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-kube-api-access-2n9v6" pod "2fb0da7d-26b1-4802-b507-b552705b9d30" (UID: "2fb0da7d-26b1-4802-b507-b552705b9d30"). InnerVolumeSpecName "kube-api-access-2n9v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:26.044922 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.044898 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-token" pod "2fb0da7d-26b1-4802-b507-b552705b9d30" (UID: "2fb0da7d-26b1-4802-b507-b552705b9d30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:26.061442 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.061409 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5tfg\" (UniqueName: \"kubernetes.io/projected/aad41311-0910-4d7f-980f-9c4062aa48c1-kube-api-access-c5tfg\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:26.111023 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.110949 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb"] Nov 03 23:06:26.121657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.121627 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs"] Nov 03 23:06:26.125096 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.124842 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:26.141121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.141088 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:26.141121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.141123 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:26.141328 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.141138 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:26.141328 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.141150 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fb0da7d-26b1-4802-b507-b552705b9d30-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:26.141328 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.141163 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n9v6\" (UniqueName: \"kubernetes.io/projected/2fb0da7d-26b1-4802-b507-b552705b9d30-kube-api-access-2n9v6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:26.315366 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.315325 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" podUID="2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:06:26.454770 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.454734 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z"] Nov 03 23:06:26.579685 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.579653 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj"] Nov 03 23:06:26.657176 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.657138 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" containerID="cri-o://39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013" gracePeriod=45 Nov 03 23:06:26.665700 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.665639 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" containerID="cri-o://7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9" gracePeriod=45 Nov 03 23:06:26.677324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.677255 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" containerID="cri-o://3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46" gracePeriod=45 Nov 03 23:06:26.758787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.758757 4903 generic.go:356] "Generic (PLEG): container finished" podID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerID="04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07" exitCode=0 Nov 03 23:06:26.758890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.758820 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" Nov 03 23:06:26.758890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.758836 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" event={"ID":"2fb0da7d-26b1-4802-b507-b552705b9d30","Type":"ContainerDied","Data":"04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07"} Nov 03 23:06:26.758890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.758885 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l" event={"ID":"2fb0da7d-26b1-4802-b507-b552705b9d30","Type":"ContainerDied","Data":"a560d9fab787a611ea5305fd2a79b56d410dad8e16781114053676d7bbe5ac6b"} Nov 03 23:06:26.759037 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.758911 4903 scope.go:122] "RemoveContainer" containerID="04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07" Nov 03 23:06:26.760499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.760470 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" event={"ID":"aad41311-0910-4d7f-980f-9c4062aa48c1","Type":"ContainerStarted","Data":"10c941d713fcad5dcd55de4e20dc4f630c67d8e54d9885f7f769e988823b9fae"} Nov 03 23:06:26.762907 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.762381 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" event={"ID":"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c","Type":"ContainerStarted","Data":"f920f84dab83a0ed132c72ae5e0c24222666d754a51d325d9f58b4a909bcfbfd"} Nov 03 23:06:26.762907 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.762435 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" event={"ID":"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c","Type":"ContainerStarted","Data":"6a11d7d79ac6b4716a8e216879537c918cfc96b66c31261fc69e229f2b52877b"} Nov 03 23:06:26.765811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.764389 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" event={"ID":"4017b29f-6cd4-484e-a0be-e97243142e4f","Type":"ContainerStarted","Data":"0e815766d0407b278c8eacec8491005575321c55b04bbe08031a3609095edd05"} Nov 03 23:06:26.765811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.764433 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" event={"ID":"4017b29f-6cd4-484e-a0be-e97243142e4f","Type":"ContainerStarted","Data":"294e7ff155ccf5c23abc5578def3f3aec6a22d22ab44caea173680bcfff291e3"} Nov 03 23:06:26.813388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.813361 4903 scope.go:122] "RemoveContainer" containerID="cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100" Nov 03 23:06:26.822867 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.822827 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs"] Nov 03 23:06:26.823731 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.823618 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" podStartSLOduration=1.8236000030000001 podStartE2EDuration="1.823600003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:26.820913264 +0000 UTC m=+3213.224854244" watchObservedRunningTime="2026-11-03 23:06:26.823600003 +0000 UTC m=+3213.227540979" Nov 03 23:06:26.823731 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.823710 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" Nov 03 23:06:26.823731 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.823731 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" Nov 03 23:06:26.823958 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.823920 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" Nov 03 23:06:26.833317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.832653 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:26.838281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.834733 4903 scope.go:122] "RemoveContainer" containerID="04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07" Nov 03 23:06:26.838281 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:26.836432 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07\": container with ID starting with 04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07 not found: ID does not exist" containerID="04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07" Nov 03 23:06:26.838281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.836473 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07"} err="failed to get container status \"04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07\": rpc error: code = NotFound desc = could not find container \"04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07\": container with ID starting with 04b4f78c52f6342bb27f64e6a4bb02a57b2f5f6bccf7fa39a1059d5955372d07 not found: ID does not exist" Nov 03 23:06:26.838281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.836499 4903 scope.go:122] "RemoveContainer" containerID="cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100" Nov 03 23:06:26.838281 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:26.837158 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100\": container with ID starting with cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100 not found: ID does not exist" containerID="cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100" Nov 03 23:06:26.838281 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.837193 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100"} err="failed to get container status \"cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100\": rpc error: code = NotFound desc = could not find container \"cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100\": container with ID starting with cafca8b930b23cfeceba83105df546c936382ff33f635a2fc4beae22ac645100 not found: ID does not exist" Nov 03 23:06:26.851210 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.851169 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs"] Nov 03 23:06:26.867379 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.864334 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" podStartSLOduration=1.864286421 podStartE2EDuration="1.864286421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:26.863505431 +0000 UTC m=+3213.267446406" watchObservedRunningTime="2026-11-03 23:06:26.864286421 +0000 UTC m=+3213.268227395" Nov 03 23:06:26.903545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.903515 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l"] Nov 03 23:06:26.909551 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.907562 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-26f8ba-openshift-default-569ccb4c87-pcd2l"] Nov 03 23:06:26.958209 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.958170 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzww8\" (UniqueName: \"kubernetes.io/projected/62e4b55c-11dc-4664-8bed-7d4dcefa05da-kube-api-access-jzww8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:26.958701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.958287 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:26.958701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.958360 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:26.999928 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:26.999902 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 23:06:27.005813 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.005727 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 23:06:27.017481 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.017459 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 23:06:27.059459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.059421 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzww8\" (UniqueName: \"kubernetes.io/projected/62e4b55c-11dc-4664-8bed-7d4dcefa05da-kube-api-access-jzww8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:27.059607 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.059510 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:27.059607 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.059586 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:27.060265 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.060239 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:27.063337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.062284 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:27.083376 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.083267 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzww8\" (UniqueName: \"kubernetes.io/projected/62e4b55c-11dc-4664-8bed-7d4dcefa05da-kube-api-access-jzww8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:27.160581 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160547 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-home\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-home\") pod \"f8c71259-4646-42b5-a78d-ae727da42f34\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " Nov 03 23:06:27.160773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160596 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-model-cache\") pod \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " Nov 03 23:06:27.160773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160627 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e2730f7-7e63-496a-9155-76818726faff-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2730f7-7e63-496a-9155-76818726faff-tls-certs\") pod \"5e2730f7-7e63-496a-9155-76818726faff\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " Nov 03 23:06:27.160773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160663 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-dshm\") pod \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " Nov 03 23:06:27.160773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160702 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-tmp-dir\") pod \"5e2730f7-7e63-496a-9155-76818726faff\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " Nov 03 23:06:27.160773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160767 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-model-cache\") pod \"5e2730f7-7e63-496a-9155-76818726faff\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " Nov 03 23:06:27.161073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160820 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-tmp-dir\") pod \"f8c71259-4646-42b5-a78d-ae727da42f34\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " Nov 03 23:06:27.161073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160857 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2730f7-7e63-496a-9155-76818726faff-kube-api-access-qks9n\" (UniqueName: \"kubernetes.io/projected/5e2730f7-7e63-496a-9155-76818726faff-kube-api-access-qks9n\") pod \"5e2730f7-7e63-496a-9155-76818726faff\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " Nov 03 23:06:27.161073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160881 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-home" pod "f8c71259-4646-42b5-a78d-ae727da42f34" (UID: "f8c71259-4646-42b5-a78d-ae727da42f34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.161073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160892 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-home\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-home\") pod \"5e2730f7-7e63-496a-9155-76818726faff\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " Nov 03 23:06:27.161073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.160997 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-model-cache" pod "9eb9ea98-1455-4f8e-b439-183fdfb7dc19" (UID: "9eb9ea98-1455-4f8e-b439-183fdfb7dc19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.161073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161005 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8c71259-4646-42b5-a78d-ae727da42f34-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c71259-4646-42b5-a78d-ae727da42f34-tls-certs\") pod \"f8c71259-4646-42b5-a78d-ae727da42f34\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " Nov 03 23:06:27.161462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161085 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-home" pod "5e2730f7-7e63-496a-9155-76818726faff" (UID: "5e2730f7-7e63-496a-9155-76818726faff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.161462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161181 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-kube-api-access-bltcx\" (UniqueName: \"kubernetes.io/projected/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-kube-api-access-bltcx\") pod \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " Nov 03 23:06:27.161462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161251 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-dshm\") pod \"5e2730f7-7e63-496a-9155-76818726faff\" (UID: \"5e2730f7-7e63-496a-9155-76818726faff\") " Nov 03 23:06:27.161462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161348 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tls-certs\") pod \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " Nov 03 23:06:27.161462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161389 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-dshm\") pod \"f8c71259-4646-42b5-a78d-ae727da42f34\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " Nov 03 23:06:27.161462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161445 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-model-cache\") pod \"f8c71259-4646-42b5-a78d-ae727da42f34\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " Nov 03 23:06:27.161795 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161503 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-home\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-home\") pod \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " Nov 03 23:06:27.161795 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161549 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c71259-4646-42b5-a78d-ae727da42f34-kube-api-access-nzglh\" (UniqueName: \"kubernetes.io/projected/f8c71259-4646-42b5-a78d-ae727da42f34-kube-api-access-nzglh\") pod \"f8c71259-4646-42b5-a78d-ae727da42f34\" (UID: \"f8c71259-4646-42b5-a78d-ae727da42f34\") " Nov 03 23:06:27.161795 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.161621 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tmp-dir\") pod \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\" (UID: \"9eb9ea98-1455-4f8e-b439-183fdfb7dc19\") " Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.162261 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.162287 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.162318 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.162357 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-tmp-dir" pod "f8c71259-4646-42b5-a78d-ae727da42f34" (UID: "f8c71259-4646-42b5-a78d-ae727da42f34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.162503 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-model-cache" pod "5e2730f7-7e63-496a-9155-76818726faff" (UID: "5e2730f7-7e63-496a-9155-76818726faff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.162528 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tmp-dir" pod "9eb9ea98-1455-4f8e-b439-183fdfb7dc19" (UID: "9eb9ea98-1455-4f8e-b439-183fdfb7dc19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.162717 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-model-cache" pod "f8c71259-4646-42b5-a78d-ae727da42f34" (UID: "f8c71259-4646-42b5-a78d-ae727da42f34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.162850 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-tmp-dir" pod "5e2730f7-7e63-496a-9155-76818726faff" (UID: "5e2730f7-7e63-496a-9155-76818726faff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.162908 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-home" pod "9eb9ea98-1455-4f8e-b439-183fdfb7dc19" (UID: "9eb9ea98-1455-4f8e-b439-183fdfb7dc19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.167271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.165266 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2730f7-7e63-496a-9155-76818726faff-tls-certs" pod "5e2730f7-7e63-496a-9155-76818726faff" (UID: "5e2730f7-7e63-496a-9155-76818726faff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:27.168232 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.167673 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c71259-4646-42b5-a78d-ae727da42f34-tls-certs" pod "f8c71259-4646-42b5-a78d-ae727da42f34" (UID: "f8c71259-4646-42b5-a78d-ae727da42f34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:27.168232 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.168095 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-dshm" pod "5e2730f7-7e63-496a-9155-76818726faff" (UID: "5e2730f7-7e63-496a-9155-76818726faff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.168823 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.168478 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tls-certs" pod "9eb9ea98-1455-4f8e-b439-183fdfb7dc19" (UID: "9eb9ea98-1455-4f8e-b439-183fdfb7dc19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:27.169216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.169195 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-dshm" pod "9eb9ea98-1455-4f8e-b439-183fdfb7dc19" (UID: "9eb9ea98-1455-4f8e-b439-183fdfb7dc19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.169370 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.169254 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2730f7-7e63-496a-9155-76818726faff-kube-api-access-qks9n" pod "5e2730f7-7e63-496a-9155-76818726faff" (UID: "5e2730f7-7e63-496a-9155-76818726faff"). InnerVolumeSpecName "kube-api-access-qks9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:27.169543 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.169509 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-kube-api-access-bltcx" pod "9eb9ea98-1455-4f8e-b439-183fdfb7dc19" (UID: "9eb9ea98-1455-4f8e-b439-183fdfb7dc19"). InnerVolumeSpecName "kube-api-access-bltcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:27.169629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.169572 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-dshm" pod "f8c71259-4646-42b5-a78d-ae727da42f34" (UID: "f8c71259-4646-42b5-a78d-ae727da42f34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.169973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.169948 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c71259-4646-42b5-a78d-ae727da42f34-kube-api-access-nzglh" pod "f8c71259-4646-42b5-a78d-ae727da42f34" (UID: "f8c71259-4646-42b5-a78d-ae727da42f34"). InnerVolumeSpecName "kube-api-access-nzglh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:27.221466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.221435 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:27.263436 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263353 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bltcx\" (UniqueName: \"kubernetes.io/projected/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-kube-api-access-bltcx\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263436 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263386 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263436 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263404 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263436 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263416 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263436 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263424 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263436 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263432 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263436 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263440 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzglh\" (UniqueName: \"kubernetes.io/projected/f8c71259-4646-42b5-a78d-ae727da42f34-kube-api-access-nzglh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263448 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263457 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e2730f7-7e63-496a-9155-76818726faff-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263464 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb9ea98-1455-4f8e-b439-183fdfb7dc19-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263471 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263478 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e2730f7-7e63-496a-9155-76818726faff-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263485 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c71259-4646-42b5-a78d-ae727da42f34-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263493 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qks9n\" (UniqueName: \"kubernetes.io/projected/5e2730f7-7e63-496a-9155-76818726faff-kube-api-access-qks9n\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.263775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.263508 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c71259-4646-42b5-a78d-ae727da42f34-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.362271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.362220 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" containerID="cri-o://6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a" gracePeriod=45 Nov 03 23:06:27.533909 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.533880 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs"] Nov 03 23:06:27.535135 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:27.535102 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e4b55c_11dc_4664_8bed_7d4dcefa05da.slice/crio-0358882988f380d14de3353f29b67c47a388ee5c74aba07135dc99da35008a1a WatchSource:0}: Error finding container 0358882988f380d14de3353f29b67c47a388ee5c74aba07135dc99da35008a1a: Status 404 returned error can't find the container with id 0358882988f380d14de3353f29b67c47a388ee5c74aba07135dc99da35008a1a Nov 03 23:06:27.617990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.617965 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 23:06:27.736769 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.736735 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" path="/var/lib/kubelet/pods/2fb0da7d-26b1-4802-b507-b552705b9d30/volumes" Nov 03 23:06:27.769558 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.769518 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-dshm\") pod \"cfc71b2b-0750-4775-ac68-9ddf63d67773\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " Nov 03 23:06:27.769692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.769595 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-tmp-dir\") pod \"cfc71b2b-0750-4775-ac68-9ddf63d67773\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " Nov 03 23:06:27.769805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.769785 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-model-cache\") pod \"cfc71b2b-0750-4775-ac68-9ddf63d67773\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " Nov 03 23:06:27.769889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.769855 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc71b2b-0750-4775-ac68-9ddf63d67773-kube-api-access-25fgj\" (UniqueName: \"kubernetes.io/projected/cfc71b2b-0750-4775-ac68-9ddf63d67773-kube-api-access-25fgj\") pod \"cfc71b2b-0750-4775-ac68-9ddf63d67773\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " Nov 03 23:06:27.769950 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.769932 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-home\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-home\") pod \"cfc71b2b-0750-4775-ac68-9ddf63d67773\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " Nov 03 23:06:27.770026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.770007 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc71b2b-0750-4775-ac68-9ddf63d67773-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc71b2b-0750-4775-ac68-9ddf63d67773-tls-certs\") pod \"cfc71b2b-0750-4775-ac68-9ddf63d67773\" (UID: \"cfc71b2b-0750-4775-ac68-9ddf63d67773\") " Nov 03 23:06:27.770233 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.770207 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-model-cache" pod "cfc71b2b-0750-4775-ac68-9ddf63d67773" (UID: "cfc71b2b-0750-4775-ac68-9ddf63d67773"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.770516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.770438 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-tmp-dir" pod "cfc71b2b-0750-4775-ac68-9ddf63d67773" (UID: "cfc71b2b-0750-4775-ac68-9ddf63d67773"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.771048 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.770693 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.771048 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.770722 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.771048 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.770930 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-home" pod "cfc71b2b-0750-4775-ac68-9ddf63d67773" (UID: "cfc71b2b-0750-4775-ac68-9ddf63d67773"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.776337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.774018 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc71b2b-0750-4775-ac68-9ddf63d67773-tls-certs" pod "cfc71b2b-0750-4775-ac68-9ddf63d67773" (UID: "cfc71b2b-0750-4775-ac68-9ddf63d67773"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:27.776337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.774057 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-dshm" pod "cfc71b2b-0750-4775-ac68-9ddf63d67773" (UID: "cfc71b2b-0750-4775-ac68-9ddf63d67773"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:27.776337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.775943 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc71b2b-0750-4775-ac68-9ddf63d67773-kube-api-access-25fgj" pod "cfc71b2b-0750-4775-ac68-9ddf63d67773" (UID: "cfc71b2b-0750-4775-ac68-9ddf63d67773"). InnerVolumeSpecName "kube-api-access-25fgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:27.780713 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.780687 4903 generic.go:356] "Generic (PLEG): container finished" podID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerID="39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013" exitCode=0 Nov 03 23:06:27.780813 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.780736 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" Nov 03 23:06:27.780813 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.780775 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" event={"ID":"9eb9ea98-1455-4f8e-b439-183fdfb7dc19","Type":"ContainerDied","Data":"39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013"} Nov 03 23:06:27.780943 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.780844 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n" event={"ID":"9eb9ea98-1455-4f8e-b439-183fdfb7dc19","Type":"ContainerDied","Data":"f6bb3a1df8be30d727d1b18d2281b47ac22a12302ebda1a7df52036c1e6affb8"} Nov 03 23:06:27.780943 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.780871 4903 scope.go:122] "RemoveContainer" containerID="39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013" Nov 03 23:06:27.783521 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.783405 4903 generic.go:356] "Generic (PLEG): container finished" podID="aad41311-0910-4d7f-980f-9c4062aa48c1" containerID="ca0eb418e76bd56f17845a2c394384ced158ceff128e43084db1bcc52c4f82c0" exitCode=0 Nov 03 23:06:27.783521 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.783485 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" event={"ID":"aad41311-0910-4d7f-980f-9c4062aa48c1","Type":"ContainerDied","Data":"ca0eb418e76bd56f17845a2c394384ced158ceff128e43084db1bcc52c4f82c0"} Nov 03 23:06:27.785253 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.785228 4903 generic.go:356] "Generic (PLEG): container finished" podID="3a6a9ff7-6e2a-420d-bc42-847c128a1b6c" containerID="f920f84dab83a0ed132c72ae5e0c24222666d754a51d325d9f58b4a909bcfbfd" exitCode=0 Nov 03 23:06:27.785397 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.785332 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" event={"ID":"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c","Type":"ContainerDied","Data":"f920f84dab83a0ed132c72ae5e0c24222666d754a51d325d9f58b4a909bcfbfd"} Nov 03 23:06:27.786886 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.786853 4903 generic.go:356] "Generic (PLEG): container finished" podID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerID="6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a" exitCode=0 Nov 03 23:06:27.786994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.786934 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" Nov 03 23:06:27.787058 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.786939 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" event={"ID":"cfc71b2b-0750-4775-ac68-9ddf63d67773","Type":"ContainerDied","Data":"6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a"} Nov 03 23:06:27.787118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.787087 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g" event={"ID":"cfc71b2b-0750-4775-ac68-9ddf63d67773","Type":"ContainerDied","Data":"e882b3e883ca0ddbb9c59a0168a1835314be333b8b9c849ce0456be890666fa5"} Nov 03 23:06:27.788330 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.788309 4903 generic.go:356] "Generic (PLEG): container finished" podID="4017b29f-6cd4-484e-a0be-e97243142e4f" containerID="0e815766d0407b278c8eacec8491005575321c55b04bbe08031a3609095edd05" exitCode=0 Nov 03 23:06:27.788447 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.788427 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" event={"ID":"4017b29f-6cd4-484e-a0be-e97243142e4f","Type":"ContainerDied","Data":"0e815766d0407b278c8eacec8491005575321c55b04bbe08031a3609095edd05"} Nov 03 23:06:27.789505 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.789486 4903 generic.go:356] "Generic (PLEG): container finished" podID="f8c71259-4646-42b5-a78d-ae727da42f34" containerID="3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46" exitCode=0 Nov 03 23:06:27.789596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.789547 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" event={"ID":"f8c71259-4646-42b5-a78d-ae727da42f34","Type":"ContainerDied","Data":"3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46"} Nov 03 23:06:27.789596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.789566 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" event={"ID":"f8c71259-4646-42b5-a78d-ae727da42f34","Type":"ContainerDied","Data":"25ab60798afdedd84038818b259a2a6a8aece89d8fc16d3b4cd433cb324a3916"} Nov 03 23:06:27.789690 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.789606 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4" Nov 03 23:06:27.791611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.791586 4903 generic.go:356] "Generic (PLEG): container finished" podID="5e2730f7-7e63-496a-9155-76818726faff" containerID="7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9" exitCode=0 Nov 03 23:06:27.791720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.791711 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" Nov 03 23:06:27.791788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.791734 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" event={"ID":"5e2730f7-7e63-496a-9155-76818726faff","Type":"ContainerDied","Data":"7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9"} Nov 03 23:06:27.791788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.791762 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8" event={"ID":"5e2730f7-7e63-496a-9155-76818726faff","Type":"ContainerDied","Data":"41bb3bac3e70f153bb28cc2a29b17bfe47f07a7cb2650b1b6424aca2c6f87a20"} Nov 03 23:06:27.793864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.793671 4903 scope.go:122] "RemoveContainer" containerID="39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013" Nov 03 23:06:27.793864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.793764 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" event={"ID":"62e4b55c-11dc-4664-8bed-7d4dcefa05da","Type":"ContainerStarted","Data":"0358882988f380d14de3353f29b67c47a388ee5c74aba07135dc99da35008a1a"} Nov 03 23:06:27.794035 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:27.794011 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013\": container with ID starting with 39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013 not found: ID does not exist" containerID="39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013" Nov 03 23:06:27.794076 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.794046 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013"} err="failed to get container status \"39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013\": rpc error: code = NotFound desc = could not find container \"39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013\": container with ID starting with 39bfa3f1f7645a6e5d190976703b2dddad8d3a43aeae42da15cacebe6e407013 not found: ID does not exist" Nov 03 23:06:27.794076 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.794071 4903 scope.go:122] "RemoveContainer" containerID="6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a" Nov 03 23:06:27.805067 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.805046 4903 scope.go:122] "RemoveContainer" containerID="6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a" Nov 03 23:06:27.805408 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:27.805386 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a\": container with ID starting with 6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a not found: ID does not exist" containerID="6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a" Nov 03 23:06:27.805523 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.805414 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a"} err="failed to get container status \"6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a\": rpc error: code = NotFound desc = could not find container \"6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a\": container with ID starting with 6c17b6ec081f1d597d8bf5e09d918dd5039dd0b73004a65ccfb55563dfa75e0a not found: ID does not exist" Nov 03 23:06:27.805523 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.805429 4903 scope.go:122] "RemoveContainer" containerID="3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46" Nov 03 23:06:27.815790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.815771 4903 scope.go:122] "RemoveContainer" containerID="3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46" Nov 03 23:06:27.816073 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:27.816054 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46\": container with ID starting with 3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46 not found: ID does not exist" containerID="3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46" Nov 03 23:06:27.816158 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.816080 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46"} err="failed to get container status \"3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46\": rpc error: code = NotFound desc = could not find container \"3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46\": container with ID starting with 3b85d6a86a7c70249d044988e6a9b1a905f982ab2ca81b1e31c87e0efed24e46 not found: ID does not exist" Nov 03 23:06:27.816158 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.816096 4903 scope.go:122] "RemoveContainer" containerID="7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9" Nov 03 23:06:27.825268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.825248 4903 scope.go:122] "RemoveContainer" containerID="7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9" Nov 03 23:06:27.825571 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:27.825550 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9\": container with ID starting with 7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9 not found: ID does not exist" containerID="7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9" Nov 03 23:06:27.825619 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.825582 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9"} err="failed to get container status \"7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9\": rpc error: code = NotFound desc = could not find container \"7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9\": container with ID starting with 7fb3cf5b1bb8cad85e94f2d1df28dafdf1e64a66af1ebc5e0200afb612f0cdb9 not found: ID does not exist" Nov 03 23:06:27.871990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.871953 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc71b2b-0750-4775-ac68-9ddf63d67773-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.871990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.871985 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.871990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.871995 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25fgj\" (UniqueName: \"kubernetes.io/projected/cfc71b2b-0750-4775-ac68-9ddf63d67773-kube-api-access-25fgj\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.872284 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.872004 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc71b2b-0750-4775-ac68-9ddf63d67773-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:27.919320 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.919281 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8"] Nov 03 23:06:27.922682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.922657 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/premium-w5-434ce3-kserve-6f6bc9669-p8fh8"] Nov 03 23:06:27.989270 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.989236 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4"] Nov 03 23:06:27.994265 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:27.994239 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/simulator-w5-434ce3-kserve-bc6bdc78d-pjfh4"] Nov 03 23:06:28.024875 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:28.024840 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n"] Nov 03 23:06:28.029187 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:28.029155 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-434ce3/distinct-w5-434ce3-kserve-794c49c7dd-dpm7n"] Nov 03 23:06:28.058491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:28.058456 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g"] Nov 03 23:06:28.062437 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:28.062412 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-50d738/simulator-w4-50d738-kserve-576c765958-mvn6g"] Nov 03 23:06:28.804469 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:28.804433 4903 generic.go:356] "Generic (PLEG): container finished" podID="62e4b55c-11dc-4664-8bed-7d4dcefa05da" containerID="aaa10baf45b65c8ffc5d64785419432ed1b94fe6401335d329ff3521c7078647" exitCode=0 Nov 03 23:06:28.804662 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:28.804520 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" event={"ID":"62e4b55c-11dc-4664-8bed-7d4dcefa05da","Type":"ContainerDied","Data":"aaa10baf45b65c8ffc5d64785419432ed1b94fe6401335d329ff3521c7078647"} Nov 03 23:06:28.979918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:28.979893 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:29.000121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.000098 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:29.015216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.015193 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:29.087627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.087544 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4017b29f-6cd4-484e-a0be-e97243142e4f-kube-api-access-mspvm\" (UniqueName: \"kubernetes.io/projected/4017b29f-6cd4-484e-a0be-e97243142e4f-kube-api-access-mspvm\") pod \"4017b29f-6cd4-484e-a0be-e97243142e4f\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " Nov 03 23:06:29.087774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.087661 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-openshift-ca\") pod \"4017b29f-6cd4-484e-a0be-e97243142e4f\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " Nov 03 23:06:29.087774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.087707 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-openshift-ca\") pod \"aad41311-0910-4d7f-980f-9c4062aa48c1\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " Nov 03 23:06:29.087906 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.087869 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-xks-ca\") pod \"4017b29f-6cd4-484e-a0be-e97243142e4f\" (UID: \"4017b29f-6cd4-484e-a0be-e97243142e4f\") " Nov 03 23:06:29.087976 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.087947 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad41311-0910-4d7f-980f-9c4062aa48c1-kube-api-access-c5tfg\" (UniqueName: \"kubernetes.io/projected/aad41311-0910-4d7f-980f-9c4062aa48c1-kube-api-access-c5tfg\") pod \"aad41311-0910-4d7f-980f-9c4062aa48c1\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " Nov 03 23:06:29.088039 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.088005 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-xks-ca\") pod \"aad41311-0910-4d7f-980f-9c4062aa48c1\" (UID: \"aad41311-0910-4d7f-980f-9c4062aa48c1\") " Nov 03 23:06:29.088039 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.088027 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-openshift-ca" pod "aad41311-0910-4d7f-980f-9c4062aa48c1" (UID: "aad41311-0910-4d7f-980f-9c4062aa48c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:29.088161 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.088071 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-openshift-ca" pod "4017b29f-6cd4-484e-a0be-e97243142e4f" (UID: "4017b29f-6cd4-484e-a0be-e97243142e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:29.088536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.088512 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:29.088536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.088532 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:29.090388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.090339 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4017b29f-6cd4-484e-a0be-e97243142e4f-kube-api-access-mspvm" pod "4017b29f-6cd4-484e-a0be-e97243142e4f" (UID: "4017b29f-6cd4-484e-a0be-e97243142e4f"). InnerVolumeSpecName "kube-api-access-mspvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:29.091142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.090403 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-xks-ca" pod "4017b29f-6cd4-484e-a0be-e97243142e4f" (UID: "4017b29f-6cd4-484e-a0be-e97243142e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:29.093257 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.093227 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad41311-0910-4d7f-980f-9c4062aa48c1-kube-api-access-c5tfg" pod "aad41311-0910-4d7f-980f-9c4062aa48c1" (UID: "aad41311-0910-4d7f-980f-9c4062aa48c1"). InnerVolumeSpecName "kube-api-access-c5tfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:29.094228 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.094199 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-xks-ca" pod "aad41311-0910-4d7f-980f-9c4062aa48c1" (UID: "aad41311-0910-4d7f-980f-9c4062aa48c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:29.189026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.188991 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-kube-api-access-2rvdh\" (UniqueName: \"kubernetes.io/projected/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-kube-api-access-2rvdh\") pod \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " Nov 03 23:06:29.189219 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.189095 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-xks-ca\") pod \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " Nov 03 23:06:29.189219 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.189148 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-openshift-ca\") pod \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\" (UID: \"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c\") " Nov 03 23:06:29.189527 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.189503 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mspvm\" (UniqueName: \"kubernetes.io/projected/4017b29f-6cd4-484e-a0be-e97243142e4f-kube-api-access-mspvm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:29.189527 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.189525 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4017b29f-6cd4-484e-a0be-e97243142e4f-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:29.189711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.189535 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5tfg\" (UniqueName: \"kubernetes.io/projected/aad41311-0910-4d7f-980f-9c4062aa48c1-kube-api-access-c5tfg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:29.189711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.189532 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-openshift-ca" pod "3a6a9ff7-6e2a-420d-bc42-847c128a1b6c" (UID: "3a6a9ff7-6e2a-420d-bc42-847c128a1b6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:29.189711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.189543 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad41311-0910-4d7f-980f-9c4062aa48c1-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:29.191184 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.191159 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-xks-ca" pod "3a6a9ff7-6e2a-420d-bc42-847c128a1b6c" (UID: "3a6a9ff7-6e2a-420d-bc42-847c128a1b6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:29.191276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.191254 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-kube-api-access-2rvdh" pod "3a6a9ff7-6e2a-420d-bc42-847c128a1b6c" (UID: "3a6a9ff7-6e2a-420d-bc42-847c128a1b6c"). InnerVolumeSpecName "kube-api-access-2rvdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:29.291156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.291124 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:29.291156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.291153 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:29.291156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.291162 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rvdh\" (UniqueName: \"kubernetes.io/projected/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c-kube-api-access-2rvdh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:29.735376 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.735336 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e2730f7-7e63-496a-9155-76818726faff" path="/var/lib/kubelet/pods/5e2730f7-7e63-496a-9155-76818726faff/volumes" Nov 03 23:06:29.735829 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.735814 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" path="/var/lib/kubelet/pods/9eb9ea98-1455-4f8e-b439-183fdfb7dc19/volumes" Nov 03 23:06:29.736213 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.736199 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" path="/var/lib/kubelet/pods/cfc71b2b-0750-4775-ac68-9ddf63d67773/volumes" Nov 03 23:06:29.736906 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.736891 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" path="/var/lib/kubelet/pods/f8c71259-4646-42b5-a78d-ae727da42f34/volumes" Nov 03 23:06:29.816100 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.816070 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" Nov 03 23:06:29.816277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.816069 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z" event={"ID":"aad41311-0910-4d7f-980f-9c4062aa48c1","Type":"ContainerDied","Data":"10c941d713fcad5dcd55de4e20dc4f630c67d8e54d9885f7f769e988823b9fae"} Nov 03 23:06:29.816277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.816195 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10c941d713fcad5dcd55de4e20dc4f630c67d8e54d9885f7f769e988823b9fae" Nov 03 23:06:29.817472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.817448 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" Nov 03 23:06:29.817596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.817486 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm" event={"ID":"3a6a9ff7-6e2a-420d-bc42-847c128a1b6c","Type":"ContainerDied","Data":"6a11d7d79ac6b4716a8e216879537c918cfc96b66c31261fc69e229f2b52877b"} Nov 03 23:06:29.817596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.817526 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a11d7d79ac6b4716a8e216879537c918cfc96b66c31261fc69e229f2b52877b" Nov 03 23:06:29.819303 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.819273 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" Nov 03 23:06:29.819407 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.819328 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw" event={"ID":"4017b29f-6cd4-484e-a0be-e97243142e4f","Type":"ContainerDied","Data":"294e7ff155ccf5c23abc5578def3f3aec6a22d22ab44caea173680bcfff291e3"} Nov 03 23:06:29.819407 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.819365 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="294e7ff155ccf5c23abc5578def3f3aec6a22d22ab44caea173680bcfff291e3" Nov 03 23:06:29.948883 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:29.948858 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:30.098930 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.098897 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-openshift-ca\") pod \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " Nov 03 23:06:30.099339 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.099029 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e4b55c-11dc-4664-8bed-7d4dcefa05da-kube-api-access-jzww8\" (UniqueName: \"kubernetes.io/projected/62e4b55c-11dc-4664-8bed-7d4dcefa05da-kube-api-access-jzww8\") pod \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " Nov 03 23:06:30.099339 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.099060 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-xks-ca\") pod \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\" (UID: \"62e4b55c-11dc-4664-8bed-7d4dcefa05da\") " Nov 03 23:06:30.099339 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.099247 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-openshift-ca" pod "62e4b55c-11dc-4664-8bed-7d4dcefa05da" (UID: "62e4b55c-11dc-4664-8bed-7d4dcefa05da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:30.099476 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.099427 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.101266 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.101240 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-xks-ca" pod "62e4b55c-11dc-4664-8bed-7d4dcefa05da" (UID: "62e4b55c-11dc-4664-8bed-7d4dcefa05da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:30.101266 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.101249 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e4b55c-11dc-4664-8bed-7d4dcefa05da-kube-api-access-jzww8" pod "62e4b55c-11dc-4664-8bed-7d4dcefa05da" (UID: "62e4b55c-11dc-4664-8bed-7d4dcefa05da"). InnerVolumeSpecName "kube-api-access-jzww8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:30.200796 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.200761 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzww8\" (UniqueName: \"kubernetes.io/projected/62e4b55c-11dc-4664-8bed-7d4dcefa05da-kube-api-access-jzww8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.200796 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.200793 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e4b55c-11dc-4664-8bed-7d4dcefa05da-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:30.829068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.829038 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" Nov 03 23:06:30.829276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.829074 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs" event={"ID":"62e4b55c-11dc-4664-8bed-7d4dcefa05da","Type":"ContainerDied","Data":"0358882988f380d14de3353f29b67c47a388ee5c74aba07135dc99da35008a1a"} Nov 03 23:06:30.829276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.829133 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0358882988f380d14de3353f29b67c47a388ee5c74aba07135dc99da35008a1a" Nov 03 23:06:30.985802 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:30.985762 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:06:31.813193 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:31.810978 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl"] Nov 03 23:06:31.909440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:31.855886 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr"] Nov 03 23:06:31.909440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:31.890967 4903 generic.go:356] "Generic (PLEG): container finished" podID="2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" containerID="f8447395ad8bc12c7623e00e428ea7e9da1253f159c963350f981e757fc741bb" exitCode=0 Nov 03 23:06:31.909440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:31.891161 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" event={"ID":"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1","Type":"ContainerDied","Data":"f8447395ad8bc12c7623e00e428ea7e9da1253f159c963350f981e757fc741bb"} Nov 03 23:06:31.909440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:31.894523 4903 generic.go:356] "Generic (PLEG): container finished" podID="84afa961-a34b-44a0-9e02-68a6dc7a04dc" containerID="1d0b761642afb5ee2225bc6bf28f11373c431eef793bee72bcf5965c72fed8f3" exitCode=0 Nov 03 23:06:31.909440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:31.894725 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" event={"ID":"84afa961-a34b-44a0-9e02-68a6dc7a04dc","Type":"ContainerDied","Data":"1d0b761642afb5ee2225bc6bf28f11373c431eef793bee72bcf5965c72fed8f3"} Nov 03 23:06:32.046503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.046103 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 23:06:32.109333 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.107783 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 23:06:32.127581 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.127498 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-plugins-config-volume\") pod \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\" (UID: \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\") " Nov 03 23:06:32.127581 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.127573 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-kube-api-access-7n88t\" (UniqueName: \"kubernetes.io/projected/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-kube-api-access-7n88t\") pod \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\" (UID: \"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1\") " Nov 03 23:06:32.127806 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.127607 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84afa961-a34b-44a0-9e02-68a6dc7a04dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afa961-a34b-44a0-9e02-68a6dc7a04dc-plugins-config-volume\") pod \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\" (UID: \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\") " Nov 03 23:06:32.127806 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.127675 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84afa961-a34b-44a0-9e02-68a6dc7a04dc-kube-api-access-tcdhk\" (UniqueName: \"kubernetes.io/projected/84afa961-a34b-44a0-9e02-68a6dc7a04dc-kube-api-access-tcdhk\") pod \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\" (UID: \"84afa961-a34b-44a0-9e02-68a6dc7a04dc\") " Nov 03 23:06:32.128070 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.128043 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84afa961-a34b-44a0-9e02-68a6dc7a04dc-plugins-config-volume" pod "84afa961-a34b-44a0-9e02-68a6dc7a04dc" (UID: "84afa961-a34b-44a0-9e02-68a6dc7a04dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:32.128499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.128472 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-plugins-config-volume" pod "2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" (UID: "2017c3ae-6dda-4b3f-bbb0-108c7b264bb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:32.130512 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.130484 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84afa961-a34b-44a0-9e02-68a6dc7a04dc-kube-api-access-tcdhk" pod "84afa961-a34b-44a0-9e02-68a6dc7a04dc" (UID: "84afa961-a34b-44a0-9e02-68a6dc7a04dc"). InnerVolumeSpecName "kube-api-access-tcdhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:32.131942 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.131917 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-kube-api-access-7n88t" pod "2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" (UID: "2017c3ae-6dda-4b3f-bbb0-108c7b264bb1"). InnerVolumeSpecName "kube-api-access-7n88t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:32.228914 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.228878 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcdhk\" (UniqueName: \"kubernetes.io/projected/84afa961-a34b-44a0-9e02-68a6dc7a04dc-kube-api-access-tcdhk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:32.228914 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.228910 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:32.228914 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.228920 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n88t\" (UniqueName: \"kubernetes.io/projected/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1-kube-api-access-7n88t\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:32.229206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.228930 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84afa961-a34b-44a0-9e02-68a6dc7a04dc-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:32.904947 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.904916 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" Nov 03 23:06:32.904947 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.904933 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk" event={"ID":"2017c3ae-6dda-4b3f-bbb0-108c7b264bb1","Type":"ContainerDied","Data":"715201807dd51dd9570d1e0b17570c40e992c709db54a3e82ff3a24e62f7c6aa"} Nov 03 23:06:32.905728 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.905016 4903 scope.go:122] "RemoveContainer" containerID="f8447395ad8bc12c7623e00e428ea7e9da1253f159c963350f981e757fc741bb" Nov 03 23:06:32.906811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.906784 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" event={"ID":"84afa961-a34b-44a0-9e02-68a6dc7a04dc","Type":"ContainerDied","Data":"d01f803632b55455c3e9795a379a96578b5b63ef450ed132afa585a8b38b4add"} Nov 03 23:06:32.907014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.906995 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl" Nov 03 23:06:32.917933 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.917816 4903 scope.go:122] "RemoveContainer" containerID="1d0b761642afb5ee2225bc6bf28f11373c431eef793bee72bcf5965c72fed8f3" Nov 03 23:06:32.967189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.967149 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk"] Nov 03 23:06:32.977890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:32.977859 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-26f8ba-6f7bb8fcbd-mkqzk"] Nov 03 23:06:33.007033 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:33.006997 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl"] Nov 03 23:06:33.013052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:33.013020 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-26f8ba-5b94f459cf-zrpcl"] Nov 03 23:06:33.064893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:33.064854 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt"] Nov 03 23:06:33.068621 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:33.068592 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-50d738-kb5nt"] Nov 03 23:06:33.736908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:33.736866 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" path="/var/lib/kubelet/pods/2017c3ae-6dda-4b3f-bbb0-108c7b264bb1/volumes" Nov 03 23:06:33.737550 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:33.737530 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224eb29b-51c4-4ce6-a564-30ce813df0d2" path="/var/lib/kubelet/pods/224eb29b-51c4-4ce6-a564-30ce813df0d2/volumes" Nov 03 23:06:33.738089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:33.738067 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84afa961-a34b-44a0-9e02-68a6dc7a04dc" path="/var/lib/kubelet/pods/84afa961-a34b-44a0-9e02-68a6dc7a04dc/volumes" Nov 03 23:06:34.551947 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.551918 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt"] Nov 03 23:06:34.552411 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552397 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" Nov 03 23:06:34.552462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552412 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" Nov 03 23:06:34.552462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552429 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4017b29f-6cd4-484e-a0be-e97243142e4f" containerName="revoke-keys" Nov 03 23:06:34.552462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552436 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4017b29f-6cd4-484e-a0be-e97243142e4f" containerName="revoke-keys" Nov 03 23:06:34.552462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552447 4903 cpu_manager.go:414] "containerMap: removing container" podUID="3a6a9ff7-6e2a-420d-bc42-847c128a1b6c" containerName="revoke-keys" Nov 03 23:06:34.552462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552452 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6a9ff7-6e2a-420d-bc42-847c128a1b6c" containerName="revoke-keys" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552464 4903 cpu_manager.go:414] "containerMap: removing container" podUID="aad41311-0910-4d7f-980f-9c4062aa48c1" containerName="revoke-keys" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552470 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad41311-0910-4d7f-980f-9c4062aa48c1" containerName="revoke-keys" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552491 4903 cpu_manager.go:414] "containerMap: removing container" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552496 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552506 4903 cpu_manager.go:414] "containerMap: removing container" podUID="62e4b55c-11dc-4664-8bed-7d4dcefa05da" containerName="revoke-keys" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552511 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e4b55c-11dc-4664-8bed-7d4dcefa05da" containerName="revoke-keys" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552522 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552526 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552538 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552543 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552551 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" containerName="payload-processing" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552556 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" containerName="payload-processing" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552568 4903 cpu_manager.go:414] "containerMap: removing container" podUID="84afa961-a34b-44a0-9e02-68a6dc7a04dc" containerName="payload-pre-processing" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552573 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84afa961-a34b-44a0-9e02-68a6dc7a04dc" containerName="payload-pre-processing" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552581 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" Nov 03 23:06:34.552634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552585 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552660 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2730f7-7e63-496a-9155-76818726faff" containerName="main" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552668 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2017c3ae-6dda-4b3f-bbb0-108c7b264bb1" containerName="payload-processing" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552677 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="84afa961-a34b-44a0-9e02-68a6dc7a04dc" containerName="payload-pre-processing" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552682 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eb9ea98-1455-4f8e-b439-183fdfb7dc19" containerName="main" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552689 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c71259-4646-42b5-a78d-ae727da42f34" containerName="main" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552694 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4017b29f-6cd4-484e-a0be-e97243142e4f" containerName="revoke-keys" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552700 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6a9ff7-6e2a-420d-bc42-847c128a1b6c" containerName="revoke-keys" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552707 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb0da7d-26b1-4802-b507-b552705b9d30" containerName="istio-proxy" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552713 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad41311-0910-4d7f-980f-9c4062aa48c1" containerName="revoke-keys" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552719 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc71b2b-0750-4775-ac68-9ddf63d67773" containerName="main" Nov 03 23:06:34.553181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.552725 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e4b55c-11dc-4664-8bed-7d4dcefa05da" containerName="revoke-keys" Nov 03 23:06:34.557682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.557662 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.564694 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.564666 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj"] Nov 03 23:06:34.568505 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.568485 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.591505 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.591476 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:34.591703 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.591485 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:34.591703 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.591506 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:06:34.591896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.591568 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:34.596961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.596941 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt"] Nov 03 23:06:34.601631 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.601604 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj"] Nov 03 23:06:34.659038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.658992 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wx4\" (UniqueName: \"kubernetes.io/projected/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-kube-api-access-g5wx4\") pod \"maas-api-revoke-sub-61aa25dc-vx5xt\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.659251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.659103 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxsm\" (UniqueName: \"kubernetes.io/projected/65e10c32-f03d-4b47-9b73-c04087642e9b-kube-api-access-pnxsm\") pod \"maas-api-revoke-sub-d4cf82f1-w6mvj\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.659251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.659149 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61aa25dc-vx5xt\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.659251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.659171 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cf82f1-w6mvj\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.659251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.659228 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cf82f1-w6mvj\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.659544 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.659374 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61aa25dc-vx5xt\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.760013 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.759974 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxsm\" (UniqueName: \"kubernetes.io/projected/65e10c32-f03d-4b47-9b73-c04087642e9b-kube-api-access-pnxsm\") pod \"maas-api-revoke-sub-d4cf82f1-w6mvj\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.760223 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.760034 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61aa25dc-vx5xt\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.760223 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.760084 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cf82f1-w6mvj\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.760223 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.760129 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cf82f1-w6mvj\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.760223 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.760213 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61aa25dc-vx5xt\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.760487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.760408 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wx4\" (UniqueName: \"kubernetes.io/projected/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-kube-api-access-g5wx4\") pod \"maas-api-revoke-sub-61aa25dc-vx5xt\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.760899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.760873 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cf82f1-w6mvj\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.760899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.760894 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61aa25dc-vx5xt\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.762708 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.762682 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cf82f1-w6mvj\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.762810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.762689 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61aa25dc-vx5xt\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.781777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.781749 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wx4\" (UniqueName: \"kubernetes.io/projected/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-kube-api-access-g5wx4\") pod \"maas-api-revoke-sub-61aa25dc-vx5xt\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.782045 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.782023 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxsm\" (UniqueName: \"kubernetes.io/projected/65e10c32-f03d-4b47-9b73-c04087642e9b-kube-api-access-pnxsm\") pod \"maas-api-revoke-sub-d4cf82f1-w6mvj\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:34.873729 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.873655 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:34.885999 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:34.885970 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:35.185143 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:35.185049 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt"] Nov 03 23:06:35.187672 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:35.187645 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c4c43b_3b7e_4b8b_8a0f_b813d655c987.slice/crio-ba8f0f50cf37c87114c26bd42b82023e6267a5134206bbb7783c4468a49c76d9 WatchSource:0}: Error finding container ba8f0f50cf37c87114c26bd42b82023e6267a5134206bbb7783c4468a49c76d9: Status 404 returned error can't find the container with id ba8f0f50cf37c87114c26bd42b82023e6267a5134206bbb7783c4468a49c76d9 Nov 03 23:06:35.200058 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:35.200030 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj"] Nov 03 23:06:35.201717 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:35.201695 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e10c32_f03d_4b47_9b73_c04087642e9b.slice/crio-d3377e24ce8d021f79c628f7c6264b22eba74ec54e209f9c7587c4c2267d5676 WatchSource:0}: Error finding container d3377e24ce8d021f79c628f7c6264b22eba74ec54e209f9c7587c4c2267d5676: Status 404 returned error can't find the container with id d3377e24ce8d021f79c628f7c6264b22eba74ec54e209f9c7587c4c2267d5676 Nov 03 23:06:35.938037 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:35.938003 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" event={"ID":"07c4c43b-3b7e-4b8b-8a0f-b813d655c987","Type":"ContainerStarted","Data":"4bfe2d0f2daf70fe01c204f539cd1ff1749ec29f20b1347a74cd32507c1e81b8"} Nov 03 23:06:35.938498 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:35.938072 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" event={"ID":"07c4c43b-3b7e-4b8b-8a0f-b813d655c987","Type":"ContainerStarted","Data":"ba8f0f50cf37c87114c26bd42b82023e6267a5134206bbb7783c4468a49c76d9"} Nov 03 23:06:35.939824 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:35.939793 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" event={"ID":"65e10c32-f03d-4b47-9b73-c04087642e9b","Type":"ContainerStarted","Data":"8601f34efd45cfc5a14c94449a501bc8cf3e57b76180849ea20727c13950b7fb"} Nov 03 23:06:35.939938 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:35.939844 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" event={"ID":"65e10c32-f03d-4b47-9b73-c04087642e9b","Type":"ContainerStarted","Data":"d3377e24ce8d021f79c628f7c6264b22eba74ec54e209f9c7587c4c2267d5676"} Nov 03 23:06:35.975962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:35.975911 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" podStartSLOduration=1.9758977359999998 podStartE2EDuration="1.975897736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:35.974497564 +0000 UTC m=+3222.378438542" watchObservedRunningTime="2026-11-03 23:06:35.975897736 +0000 UTC m=+3222.379838739" Nov 03 23:06:36.011938 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.011880 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" podStartSLOduration=2.01186214 podStartE2EDuration="2.01186214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:06:36.010012738 +0000 UTC m=+3222.413953713" watchObservedRunningTime="2026-11-03 23:06:36.01186214 +0000 UTC m=+3222.415803128" Nov 03 23:06:36.213200 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.213121 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz"] Nov 03 23:06:36.241336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.241273 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv"] Nov 03 23:06:36.812588 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.812523 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" podUID="76294da7-a114-4504-8530-a7bb4b3136be" containerName="payload-processing" containerID="cri-o://88034564e11c074b09a9c81730153d0823e3080e3eae8431b19e84ea9f76db8e" gracePeriod=25 Nov 03 23:06:36.857982 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.857907 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" podUID="5f387f8d-1889-4269-b3b2-29fe3ba25667" containerName="payload-pre-processing" containerID="cri-o://72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35" gracePeriod=25 Nov 03 23:06:36.950374 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.950335 4903 generic.go:356] "Generic (PLEG): container finished" podID="07c4c43b-3b7e-4b8b-8a0f-b813d655c987" containerID="4bfe2d0f2daf70fe01c204f539cd1ff1749ec29f20b1347a74cd32507c1e81b8" exitCode=0 Nov 03 23:06:36.950808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.950440 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" event={"ID":"07c4c43b-3b7e-4b8b-8a0f-b813d655c987","Type":"ContainerDied","Data":"4bfe2d0f2daf70fe01c204f539cd1ff1749ec29f20b1347a74cd32507c1e81b8"} Nov 03 23:06:36.952273 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.952249 4903 generic.go:356] "Generic (PLEG): container finished" podID="65e10c32-f03d-4b47-9b73-c04087642e9b" containerID="8601f34efd45cfc5a14c94449a501bc8cf3e57b76180849ea20727c13950b7fb" exitCode=0 Nov 03 23:06:36.952415 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.952333 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" event={"ID":"65e10c32-f03d-4b47-9b73-c04087642e9b","Type":"ContainerDied","Data":"8601f34efd45cfc5a14c94449a501bc8cf3e57b76180849ea20727c13950b7fb"} Nov 03 23:06:36.953522 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.953501 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq"] Nov 03 23:06:36.954026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.954009 4903 generic.go:356] "Generic (PLEG): container finished" podID="76294da7-a114-4504-8530-a7bb4b3136be" containerID="88034564e11c074b09a9c81730153d0823e3080e3eae8431b19e84ea9f76db8e" exitCode=0 Nov 03 23:06:36.954102 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.954076 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" event={"ID":"76294da7-a114-4504-8530-a7bb4b3136be","Type":"ContainerDied","Data":"88034564e11c074b09a9c81730153d0823e3080e3eae8431b19e84ea9f76db8e"} Nov 03 23:06:36.960215 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:36.960192 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-434ce3-sd4vq"] Nov 03 23:06:37.135311 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.135262 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 23:06:37.169154 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.169124 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 23:06:37.187608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.187574 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76294da7-a114-4504-8530-a7bb4b3136be-kube-api-access-cmm5l\" (UniqueName: \"kubernetes.io/projected/76294da7-a114-4504-8530-a7bb4b3136be-kube-api-access-cmm5l\") pod \"76294da7-a114-4504-8530-a7bb4b3136be\" (UID: \"76294da7-a114-4504-8530-a7bb4b3136be\") " Nov 03 23:06:37.188072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.188044 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76294da7-a114-4504-8530-a7bb4b3136be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76294da7-a114-4504-8530-a7bb4b3136be-plugins-config-volume\") pod \"76294da7-a114-4504-8530-a7bb4b3136be\" (UID: \"76294da7-a114-4504-8530-a7bb4b3136be\") " Nov 03 23:06:37.188340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.188285 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f387f8d-1889-4269-b3b2-29fe3ba25667-kube-api-access-lksb4\" (UniqueName: \"kubernetes.io/projected/5f387f8d-1889-4269-b3b2-29fe3ba25667-kube-api-access-lksb4\") pod \"5f387f8d-1889-4269-b3b2-29fe3ba25667\" (UID: \"5f387f8d-1889-4269-b3b2-29fe3ba25667\") " Nov 03 23:06:37.188814 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.188504 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f387f8d-1889-4269-b3b2-29fe3ba25667-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f387f8d-1889-4269-b3b2-29fe3ba25667-plugins-config-volume\") pod \"5f387f8d-1889-4269-b3b2-29fe3ba25667\" (UID: \"5f387f8d-1889-4269-b3b2-29fe3ba25667\") " Nov 03 23:06:37.189243 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.188919 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" podUID="08a1b497-df74-4ee0-aa16-447d4a1f4c06" containerName="main" containerID="cri-o://c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0" gracePeriod=45 Nov 03 23:06:37.189598 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.189547 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76294da7-a114-4504-8530-a7bb4b3136be-plugins-config-volume" pod "76294da7-a114-4504-8530-a7bb4b3136be" (UID: "76294da7-a114-4504-8530-a7bb4b3136be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:37.189841 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.189818 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f387f8d-1889-4269-b3b2-29fe3ba25667-plugins-config-volume" pod "5f387f8d-1889-4269-b3b2-29fe3ba25667" (UID: "5f387f8d-1889-4269-b3b2-29fe3ba25667"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:37.190324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.190271 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76294da7-a114-4504-8530-a7bb4b3136be-kube-api-access-cmm5l" pod "76294da7-a114-4504-8530-a7bb4b3136be" (UID: "76294da7-a114-4504-8530-a7bb4b3136be"). InnerVolumeSpecName "kube-api-access-cmm5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:37.191475 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.191443 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f387f8d-1889-4269-b3b2-29fe3ba25667-kube-api-access-lksb4" pod "5f387f8d-1889-4269-b3b2-29fe3ba25667" (UID: "5f387f8d-1889-4269-b3b2-29fe3ba25667"). InnerVolumeSpecName "kube-api-access-lksb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:37.289927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.289889 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f387f8d-1889-4269-b3b2-29fe3ba25667-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.289927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.289920 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmm5l\" (UniqueName: \"kubernetes.io/projected/76294da7-a114-4504-8530-a7bb4b3136be-kube-api-access-cmm5l\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.289927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.289930 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76294da7-a114-4504-8530-a7bb4b3136be-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.289927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.289939 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lksb4\" (UniqueName: \"kubernetes.io/projected/5f387f8d-1889-4269-b3b2-29fe3ba25667-kube-api-access-lksb4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.463309 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.463276 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:06:37.492059 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492028 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tls-certs\" (UniqueName: \"kubernetes.io/secret/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tls-certs\") pod \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " Nov 03 23:06:37.492244 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492078 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tmp-dir\") pod \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " Nov 03 23:06:37.492244 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492102 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-model-cache\") pod \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " Nov 03 23:06:37.492244 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492144 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-dshm\") pod \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " Nov 03 23:06:37.492244 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492168 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a1b497-df74-4ee0-aa16-447d4a1f4c06-kube-api-access-56wl9\" (UniqueName: \"kubernetes.io/projected/08a1b497-df74-4ee0-aa16-447d4a1f4c06-kube-api-access-56wl9\") pod \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " Nov 03 23:06:37.492506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492394 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-home\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-home\") pod \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\" (UID: \"08a1b497-df74-4ee0-aa16-447d4a1f4c06\") " Nov 03 23:06:37.492506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492413 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tmp-dir" pod "08a1b497-df74-4ee0-aa16-447d4a1f4c06" (UID: "08a1b497-df74-4ee0-aa16-447d4a1f4c06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:37.492506 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492433 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-model-cache" pod "08a1b497-df74-4ee0-aa16-447d4a1f4c06" (UID: "08a1b497-df74-4ee0-aa16-447d4a1f4c06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:37.492693 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492672 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-home" pod "08a1b497-df74-4ee0-aa16-447d4a1f4c06" (UID: "08a1b497-df74-4ee0-aa16-447d4a1f4c06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:37.492893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492873 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.492948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492899 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.492948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.492913 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.494509 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.494481 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-dshm" pod "08a1b497-df74-4ee0-aa16-447d4a1f4c06" (UID: "08a1b497-df74-4ee0-aa16-447d4a1f4c06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:37.494617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.494533 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tls-certs" pod "08a1b497-df74-4ee0-aa16-447d4a1f4c06" (UID: "08a1b497-df74-4ee0-aa16-447d4a1f4c06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:37.494617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.494540 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a1b497-df74-4ee0-aa16-447d4a1f4c06-kube-api-access-56wl9" pod "08a1b497-df74-4ee0-aa16-447d4a1f4c06" (UID: "08a1b497-df74-4ee0-aa16-447d4a1f4c06"). InnerVolumeSpecName "kube-api-access-56wl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:37.593899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.593867 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a1b497-df74-4ee0-aa16-447d4a1f4c06-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.593899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.593895 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a1b497-df74-4ee0-aa16-447d4a1f4c06-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.593899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.593904 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56wl9\" (UniqueName: \"kubernetes.io/projected/08a1b497-df74-4ee0-aa16-447d4a1f4c06-kube-api-access-56wl9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:37.740024 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.739988 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bedbab-6ddd-4a91-bd60-38d799fa8e16" path="/var/lib/kubelet/pods/e5bedbab-6ddd-4a91-bd60-38d799fa8e16/volumes" Nov 03 23:06:37.963915 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.963882 4903 generic.go:356] "Generic (PLEG): container finished" podID="08a1b497-df74-4ee0-aa16-447d4a1f4c06" containerID="c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0" exitCode=0 Nov 03 23:06:37.964383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.963940 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" Nov 03 23:06:37.964383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.963976 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" event={"ID":"08a1b497-df74-4ee0-aa16-447d4a1f4c06","Type":"ContainerDied","Data":"c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0"} Nov 03 23:06:37.964383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.964023 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75" event={"ID":"08a1b497-df74-4ee0-aa16-447d4a1f4c06","Type":"ContainerDied","Data":"793765f5122fbce2eb5998f43841429cfea5cfdf669eae1e093ff2c8fafbbbae"} Nov 03 23:06:37.964383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.964041 4903 scope.go:122] "RemoveContainer" containerID="c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0" Nov 03 23:06:37.965601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.965578 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" event={"ID":"76294da7-a114-4504-8530-a7bb4b3136be","Type":"ContainerDied","Data":"94242b1a0704c9feb5bc78272aafc60a3255ad060acf76944546840099599142"} Nov 03 23:06:37.965685 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.965598 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl" Nov 03 23:06:37.967066 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.967041 4903 generic.go:356] "Generic (PLEG): container finished" podID="5f387f8d-1889-4269-b3b2-29fe3ba25667" containerID="72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35" exitCode=0 Nov 03 23:06:37.967157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.967106 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" Nov 03 23:06:37.967157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.967127 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" event={"ID":"5f387f8d-1889-4269-b3b2-29fe3ba25667","Type":"ContainerDied","Data":"72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35"} Nov 03 23:06:37.967227 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.967166 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr" event={"ID":"5f387f8d-1889-4269-b3b2-29fe3ba25667","Type":"ContainerDied","Data":"7cb0561ad861b93bde63d6e62693d9a0193252cc6aa369e0a0e93e319584f326"} Nov 03 23:06:37.977078 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.977039 4903 scope.go:122] "RemoveContainer" containerID="c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0" Nov 03 23:06:37.977392 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:37.977371 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0\": container with ID starting with c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0 not found: ID does not exist" containerID="c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0" Nov 03 23:06:37.977463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.977398 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0"} err="failed to get container status \"c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0\": rpc error: code = NotFound desc = could not find container \"c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0\": container with ID starting with c6bebc82a08669b80c312e0712b88340d4d38f49b02b9f50ea44e453faf036b0 not found: ID does not exist" Nov 03 23:06:37.977463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.977417 4903 scope.go:122] "RemoveContainer" containerID="88034564e11c074b09a9c81730153d0823e3080e3eae8431b19e84ea9f76db8e" Nov 03 23:06:37.989320 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:37.989255 4903 scope.go:122] "RemoveContainer" containerID="72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35" Nov 03 23:06:38.003503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.003481 4903 scope.go:122] "RemoveContainer" containerID="72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35" Nov 03 23:06:38.003862 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:38.003842 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35\": container with ID starting with 72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35 not found: ID does not exist" containerID="72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35" Nov 03 23:06:38.003938 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.003876 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35"} err="failed to get container status \"72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35\": rpc error: code = NotFound desc = could not find container \"72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35\": container with ID starting with 72db968b3a106cfb08bd0bc5c00df32bcf8851fd26f0e69572ccb15422801c35 not found: ID does not exist" Nov 03 23:06:38.017176 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.017122 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl"] Nov 03 23:06:38.021645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.021614 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2d619f-bdbc76d94-99hjl"] Nov 03 23:06:38.052608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.051826 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75"] Nov 03 23:06:38.055735 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.055708 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-23d281/rate-test-model-23d281-kserve-78865749-htr75"] Nov 03 23:06:38.096642 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.096603 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr"] Nov 03 23:06:38.101798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.101768 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-2d619f-5c9fd58999-69tjr"] Nov 03 23:06:38.151149 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.151110 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:38.187311 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.187276 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:38.207833 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.207804 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-openshift-ca\") pod \"65e10c32-f03d-4b47-9b73-c04087642e9b\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " Nov 03 23:06:38.208006 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.207844 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-xks-ca\") pod \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " Nov 03 23:06:38.208006 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.207919 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-xks-ca\") pod \"65e10c32-f03d-4b47-9b73-c04087642e9b\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " Nov 03 23:06:38.208006 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.207956 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e10c32-f03d-4b47-9b73-c04087642e9b-kube-api-access-pnxsm\" (UniqueName: \"kubernetes.io/projected/65e10c32-f03d-4b47-9b73-c04087642e9b-kube-api-access-pnxsm\") pod \"65e10c32-f03d-4b47-9b73-c04087642e9b\" (UID: \"65e10c32-f03d-4b47-9b73-c04087642e9b\") " Nov 03 23:06:38.208181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.208008 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-kube-api-access-g5wx4\" (UniqueName: \"kubernetes.io/projected/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-kube-api-access-g5wx4\") pod \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " Nov 03 23:06:38.208181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.208122 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-openshift-ca\") pod \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\" (UID: \"07c4c43b-3b7e-4b8b-8a0f-b813d655c987\") " Nov 03 23:06:38.208329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.208202 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-openshift-ca" pod "65e10c32-f03d-4b47-9b73-c04087642e9b" (UID: "65e10c32-f03d-4b47-9b73-c04087642e9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:38.208613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.208588 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:38.208751 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.208718 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-openshift-ca" pod "07c4c43b-3b7e-4b8b-8a0f-b813d655c987" (UID: "07c4c43b-3b7e-4b8b-8a0f-b813d655c987"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:38.210729 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.210702 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-xks-ca" pod "07c4c43b-3b7e-4b8b-8a0f-b813d655c987" (UID: "07c4c43b-3b7e-4b8b-8a0f-b813d655c987"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:38.210966 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.210937 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e10c32-f03d-4b47-9b73-c04087642e9b-kube-api-access-pnxsm" pod "65e10c32-f03d-4b47-9b73-c04087642e9b" (UID: "65e10c32-f03d-4b47-9b73-c04087642e9b"). InnerVolumeSpecName "kube-api-access-pnxsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:38.214923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.214894 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-xks-ca" pod "65e10c32-f03d-4b47-9b73-c04087642e9b" (UID: "65e10c32-f03d-4b47-9b73-c04087642e9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:38.218465 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.218443 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-kube-api-access-g5wx4" pod "07c4c43b-3b7e-4b8b-8a0f-b813d655c987" (UID: "07c4c43b-3b7e-4b8b-8a0f-b813d655c987"). InnerVolumeSpecName "kube-api-access-g5wx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:38.309994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.309919 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:38.309994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.309948 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:38.309994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.309957 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10c32-f03d-4b47-9b73-c04087642e9b-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:38.309994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.309966 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxsm\" (UniqueName: \"kubernetes.io/projected/65e10c32-f03d-4b47-9b73-c04087642e9b-kube-api-access-pnxsm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:38.309994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.309975 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5wx4\" (UniqueName: \"kubernetes.io/projected/07c4c43b-3b7e-4b8b-8a0f-b813d655c987-kube-api-access-g5wx4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:38.976814 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.976778 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" Nov 03 23:06:38.977284 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.976779 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt" event={"ID":"07c4c43b-3b7e-4b8b-8a0f-b813d655c987","Type":"ContainerDied","Data":"ba8f0f50cf37c87114c26bd42b82023e6267a5134206bbb7783c4468a49c76d9"} Nov 03 23:06:38.977284 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.976927 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba8f0f50cf37c87114c26bd42b82023e6267a5134206bbb7783c4468a49c76d9" Nov 03 23:06:38.978340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.978289 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" Nov 03 23:06:38.978340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.978325 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj" event={"ID":"65e10c32-f03d-4b47-9b73-c04087642e9b","Type":"ContainerDied","Data":"d3377e24ce8d021f79c628f7c6264b22eba74ec54e209f9c7587c4c2267d5676"} Nov 03 23:06:38.978544 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:38.978376 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3377e24ce8d021f79c628f7c6264b22eba74ec54e209f9c7587c4c2267d5676" Nov 03 23:06:39.733171 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:39.733137 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a1b497-df74-4ee0-aa16-447d4a1f4c06" path="/var/lib/kubelet/pods/08a1b497-df74-4ee0-aa16-447d4a1f4c06/volumes" Nov 03 23:06:39.733587 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:39.733571 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f387f8d-1889-4269-b3b2-29fe3ba25667" path="/var/lib/kubelet/pods/5f387f8d-1889-4269-b3b2-29fe3ba25667/volumes" Nov 03 23:06:39.733963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:39.733949 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76294da7-a114-4504-8530-a7bb4b3136be" path="/var/lib/kubelet/pods/76294da7-a114-4504-8530-a7bb4b3136be/volumes" Nov 03 23:06:40.806350 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:40.806315 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs"] Nov 03 23:06:40.816402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:40.816374 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-23d281-sflbs"] Nov 03 23:06:41.136136 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.136050 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" containerID="cri-o://8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2" gracePeriod=45 Nov 03 23:06:41.153168 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.153125 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" containerID="cri-o://f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3" gracePeriod=45 Nov 03 23:06:41.214286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.214229 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" podUID="1c106a89-020e-4116-9077-19f6304b4685" containerName="payload-processing" containerID="cri-o://77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe" gracePeriod=25 Nov 03 23:06:41.241902 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.241826 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" podUID="eb3f5424-1ea1-4178-88e3-64b8f00bf1fd" containerName="payload-pre-processing" containerID="cri-o://a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a" gracePeriod=25 Nov 03 23:06:41.417376 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.413466 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Nov 03 23:06:41.540541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.540507 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 23:06:41.593117 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.593088 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 23:06:41.598407 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.598378 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 23:06:41.600821 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.600788 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" podUID="7d08ad03-a13d-4b82-a25d-00f821e29725" containerName="main" containerID="cri-o://96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd" gracePeriod=45 Nov 03 23:06:41.623954 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.623927 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 23:06:41.664429 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664395 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-plugins-config-volume\") pod \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\" (UID: \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\") " Nov 03 23:06:41.664601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664448 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tmp-dir\") pod \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " Nov 03 23:06:41.664601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664524 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-dshm\") pod \"599f7b09-4f28-499c-bb80-cf9811e68a73\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " Nov 03 23:06:41.664601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664561 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-model-cache\") pod \"599f7b09-4f28-499c-bb80-cf9811e68a73\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " Nov 03 23:06:41.664805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664619 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c106a89-020e-4116-9077-19f6304b4685-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c106a89-020e-4116-9077-19f6304b4685-plugins-config-volume\") pod \"1c106a89-020e-4116-9077-19f6304b4685\" (UID: \"1c106a89-020e-4116-9077-19f6304b4685\") " Nov 03 23:06:41.664805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664673 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-model-cache\") pod \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " Nov 03 23:06:41.664805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664703 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599f7b09-4f28-499c-bb80-cf9811e68a73-kube-api-access-xdfgp\" (UniqueName: \"kubernetes.io/projected/599f7b09-4f28-499c-bb80-cf9811e68a73-kube-api-access-xdfgp\") pod \"599f7b09-4f28-499c-bb80-cf9811e68a73\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " Nov 03 23:06:41.664805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664764 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-tmp-dir\") pod \"599f7b09-4f28-499c-bb80-cf9811e68a73\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " Nov 03 23:06:41.664805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664797 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c106a89-020e-4116-9077-19f6304b4685-kube-api-access-g5l9q\" (UniqueName: \"kubernetes.io/projected/1c106a89-020e-4116-9077-19f6304b4685-kube-api-access-g5l9q\") pod \"1c106a89-020e-4116-9077-19f6304b4685\" (UID: \"1c106a89-020e-4116-9077-19f6304b4685\") " Nov 03 23:06:41.665081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664821 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-plugins-config-volume" pod "eb3f5424-1ea1-4178-88e3-64b8f00bf1fd" (UID: "eb3f5424-1ea1-4178-88e3-64b8f00bf1fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:41.665081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664887 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tls-certs\") pod \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " Nov 03 23:06:41.665081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664934 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-home\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-home\") pod \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " Nov 03 23:06:41.665081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.664984 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-dshm\") pod \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " Nov 03 23:06:41.665081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.665028 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-home\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-home\") pod \"599f7b09-4f28-499c-bb80-cf9811e68a73\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " Nov 03 23:06:41.665081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.665066 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599f7b09-4f28-499c-bb80-cf9811e68a73-tls-certs\" (UniqueName: \"kubernetes.io/secret/599f7b09-4f28-499c-bb80-cf9811e68a73-tls-certs\") pod \"599f7b09-4f28-499c-bb80-cf9811e68a73\" (UID: \"599f7b09-4f28-499c-bb80-cf9811e68a73\") " Nov 03 23:06:41.665633 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.665100 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-kube-api-access-4v2q6\" (UniqueName: \"kubernetes.io/projected/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-kube-api-access-4v2q6\") pod \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\" (UID: \"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd\") " Nov 03 23:06:41.665633 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.665133 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-kube-api-access-8jvrd\" (UniqueName: \"kubernetes.io/projected/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-kube-api-access-8jvrd\") pod \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\" (UID: \"806bbbeb-b428-44e2-95c1-4cf3282eb1cd\") " Nov 03 23:06:41.665633 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.665259 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c106a89-020e-4116-9077-19f6304b4685-plugins-config-volume" pod "1c106a89-020e-4116-9077-19f6304b4685" (UID: "1c106a89-020e-4116-9077-19f6304b4685"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:41.665788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.665742 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c106a89-020e-4116-9077-19f6304b4685-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.665788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.665766 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.665788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.665368 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tmp-dir" pod "806bbbeb-b428-44e2-95c1-4cf3282eb1cd" (UID: "806bbbeb-b428-44e2-95c1-4cf3282eb1cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.665963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.665852 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-model-cache" pod "599f7b09-4f28-499c-bb80-cf9811e68a73" (UID: "599f7b09-4f28-499c-bb80-cf9811e68a73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.666094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.666068 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-home" pod "599f7b09-4f28-499c-bb80-cf9811e68a73" (UID: "599f7b09-4f28-499c-bb80-cf9811e68a73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.666637 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.666497 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-tmp-dir" pod "599f7b09-4f28-499c-bb80-cf9811e68a73" (UID: "599f7b09-4f28-499c-bb80-cf9811e68a73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.666824 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.666785 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-model-cache" pod "806bbbeb-b428-44e2-95c1-4cf3282eb1cd" (UID: "806bbbeb-b428-44e2-95c1-4cf3282eb1cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.666919 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.666836 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-home" pod "806bbbeb-b428-44e2-95c1-4cf3282eb1cd" (UID: "806bbbeb-b428-44e2-95c1-4cf3282eb1cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.668628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.668567 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-dshm" pod "599f7b09-4f28-499c-bb80-cf9811e68a73" (UID: "599f7b09-4f28-499c-bb80-cf9811e68a73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.670363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.668837 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-kube-api-access-8jvrd" pod "806bbbeb-b428-44e2-95c1-4cf3282eb1cd" (UID: "806bbbeb-b428-44e2-95c1-4cf3282eb1cd"). InnerVolumeSpecName "kube-api-access-8jvrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:41.670363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.669447 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-kube-api-access-4v2q6" pod "eb3f5424-1ea1-4178-88e3-64b8f00bf1fd" (UID: "eb3f5424-1ea1-4178-88e3-64b8f00bf1fd"). InnerVolumeSpecName "kube-api-access-4v2q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:41.670363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.669934 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599f7b09-4f28-499c-bb80-cf9811e68a73-kube-api-access-xdfgp" pod "599f7b09-4f28-499c-bb80-cf9811e68a73" (UID: "599f7b09-4f28-499c-bb80-cf9811e68a73"). InnerVolumeSpecName "kube-api-access-xdfgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:41.670363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.670179 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tls-certs" pod "806bbbeb-b428-44e2-95c1-4cf3282eb1cd" (UID: "806bbbeb-b428-44e2-95c1-4cf3282eb1cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:41.671368 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.671339 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c106a89-020e-4116-9077-19f6304b4685-kube-api-access-g5l9q" pod "1c106a89-020e-4116-9077-19f6304b4685" (UID: "1c106a89-020e-4116-9077-19f6304b4685"). InnerVolumeSpecName "kube-api-access-g5l9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:41.672554 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.672527 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599f7b09-4f28-499c-bb80-cf9811e68a73-tls-certs" pod "599f7b09-4f28-499c-bb80-cf9811e68a73" (UID: "599f7b09-4f28-499c-bb80-cf9811e68a73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:41.676463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.675600 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-dshm" pod "806bbbeb-b428-44e2-95c1-4cf3282eb1cd" (UID: "806bbbeb-b428-44e2-95c1-4cf3282eb1cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.757708 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.757659 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e4b55c-11dc-4664-8bed-7d4dcefa05da" path="/var/lib/kubelet/pods/62e4b55c-11dc-4664-8bed-7d4dcefa05da/volumes" Nov 03 23:06:41.768449 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.768417 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v2q6\" (UniqueName: \"kubernetes.io/projected/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd-kube-api-access-4v2q6\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.768679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.768663 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jvrd\" (UniqueName: \"kubernetes.io/projected/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-kube-api-access-8jvrd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.768789 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.768774 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.768882 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.768872 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.768962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.768952 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.769044 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.769034 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.769130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.769120 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdfgp\" (UniqueName: \"kubernetes.io/projected/599f7b09-4f28-499c-bb80-cf9811e68a73-kube-api-access-xdfgp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.769211 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.769200 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.769355 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.769342 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5l9q\" (UniqueName: \"kubernetes.io/projected/1c106a89-020e-4116-9077-19f6304b4685-kube-api-access-g5l9q\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.769450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.769439 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.769541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.769532 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.769631 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.769621 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/806bbbeb-b428-44e2-95c1-4cf3282eb1cd-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.769715 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.769705 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/599f7b09-4f28-499c-bb80-cf9811e68a73-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.769795 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.769786 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/599f7b09-4f28-499c-bb80-cf9811e68a73-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.881004 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.880979 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:06:41.972362 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.972253 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-home\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-home\") pod \"7d08ad03-a13d-4b82-a25d-00f821e29725\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " Nov 03 23:06:41.972362 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.972332 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-model-cache\") pod \"7d08ad03-a13d-4b82-a25d-00f821e29725\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " Nov 03 23:06:41.972595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.972388 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d08ad03-a13d-4b82-a25d-00f821e29725-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d08ad03-a13d-4b82-a25d-00f821e29725-tls-certs\") pod \"7d08ad03-a13d-4b82-a25d-00f821e29725\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " Nov 03 23:06:41.972595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.972441 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d08ad03-a13d-4b82-a25d-00f821e29725-kube-api-access-rpwhq\" (UniqueName: \"kubernetes.io/projected/7d08ad03-a13d-4b82-a25d-00f821e29725-kube-api-access-rpwhq\") pod \"7d08ad03-a13d-4b82-a25d-00f821e29725\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " Nov 03 23:06:41.972595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.972490 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-dshm\") pod \"7d08ad03-a13d-4b82-a25d-00f821e29725\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " Nov 03 23:06:41.972595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.972549 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-home" pod "7d08ad03-a13d-4b82-a25d-00f821e29725" (UID: "7d08ad03-a13d-4b82-a25d-00f821e29725"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.972595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.972562 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-tmp-dir\") pod \"7d08ad03-a13d-4b82-a25d-00f821e29725\" (UID: \"7d08ad03-a13d-4b82-a25d-00f821e29725\") " Nov 03 23:06:41.972871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.972589 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-model-cache" pod "7d08ad03-a13d-4b82-a25d-00f821e29725" (UID: "7d08ad03-a13d-4b82-a25d-00f821e29725"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.972871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.972803 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-tmp-dir" pod "7d08ad03-a13d-4b82-a25d-00f821e29725" (UID: "7d08ad03-a13d-4b82-a25d-00f821e29725"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.973062 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.973039 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.973062 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.973060 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.973253 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.973070 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:41.974684 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.974658 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-dshm" pod "7d08ad03-a13d-4b82-a25d-00f821e29725" (UID: "7d08ad03-a13d-4b82-a25d-00f821e29725"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:41.974774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.974694 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d08ad03-a13d-4b82-a25d-00f821e29725-tls-certs" pod "7d08ad03-a13d-4b82-a25d-00f821e29725" (UID: "7d08ad03-a13d-4b82-a25d-00f821e29725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:41.974774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:41.974747 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d08ad03-a13d-4b82-a25d-00f821e29725-kube-api-access-rpwhq" pod "7d08ad03-a13d-4b82-a25d-00f821e29725" (UID: "7d08ad03-a13d-4b82-a25d-00f821e29725"). InnerVolumeSpecName "kube-api-access-rpwhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:42.005227 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.005195 4903 generic.go:356] "Generic (PLEG): container finished" podID="1c106a89-020e-4116-9077-19f6304b4685" containerID="77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe" exitCode=0 Nov 03 23:06:42.005410 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.005262 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" Nov 03 23:06:42.005410 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.005275 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" event={"ID":"1c106a89-020e-4116-9077-19f6304b4685","Type":"ContainerDied","Data":"77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe"} Nov 03 23:06:42.005410 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.005358 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz" event={"ID":"1c106a89-020e-4116-9077-19f6304b4685","Type":"ContainerDied","Data":"35c6e1f39f800731cac0429f0ec0cc5ff3bb0f68a3a2e01a9c9e5c5f842b551a"} Nov 03 23:06:42.005410 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.005382 4903 scope.go:122] "RemoveContainer" containerID="77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe" Nov 03 23:06:42.006798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.006776 4903 generic.go:356] "Generic (PLEG): container finished" podID="7d08ad03-a13d-4b82-a25d-00f821e29725" containerID="96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd" exitCode=0 Nov 03 23:06:42.006912 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.006817 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" Nov 03 23:06:42.006912 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.006836 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" event={"ID":"7d08ad03-a13d-4b82-a25d-00f821e29725","Type":"ContainerDied","Data":"96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd"} Nov 03 23:06:42.006912 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.006872 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj" event={"ID":"7d08ad03-a13d-4b82-a25d-00f821e29725","Type":"ContainerDied","Data":"f577352653e1877050dbabfb4e6f20bac371dc105d5b3bba9d873c2218286ae2"} Nov 03 23:06:42.008724 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.008700 4903 generic.go:356] "Generic (PLEG): container finished" podID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerID="8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2" exitCode=0 Nov 03 23:06:42.008834 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.008773 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" Nov 03 23:06:42.008908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.008770 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" event={"ID":"599f7b09-4f28-499c-bb80-cf9811e68a73","Type":"ContainerDied","Data":"8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2"} Nov 03 23:06:42.008996 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.008903 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb" event={"ID":"599f7b09-4f28-499c-bb80-cf9811e68a73","Type":"ContainerDied","Data":"a73ebd7e5d0f732c34ab145628ac7fd0854db82a175a860a1d23c86d1a14d241"} Nov 03 23:06:42.010481 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.010459 4903 generic.go:356] "Generic (PLEG): container finished" podID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerID="f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3" exitCode=0 Nov 03 23:06:42.010672 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.010499 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" event={"ID":"806bbbeb-b428-44e2-95c1-4cf3282eb1cd","Type":"ContainerDied","Data":"f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3"} Nov 03 23:06:42.010672 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.010513 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" Nov 03 23:06:42.010672 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.010529 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs" event={"ID":"806bbbeb-b428-44e2-95c1-4cf3282eb1cd","Type":"ContainerDied","Data":"979e7e2179fe59545b5bd078b8a298fb1d8bd080f01898a25372b0b908b4f47b"} Nov 03 23:06:42.012392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.012364 4903 generic.go:356] "Generic (PLEG): container finished" podID="eb3f5424-1ea1-4178-88e3-64b8f00bf1fd" containerID="a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a" exitCode=0 Nov 03 23:06:42.012791 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.012409 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" Nov 03 23:06:42.012791 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.012448 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" event={"ID":"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd","Type":"ContainerDied","Data":"a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a"} Nov 03 23:06:42.012791 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.012487 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv" event={"ID":"eb3f5424-1ea1-4178-88e3-64b8f00bf1fd","Type":"ContainerDied","Data":"d01c7ac47a7c84dc5b83032fd987506e536d326201f507559772b5dc16b41024"} Nov 03 23:06:42.018157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.018136 4903 scope.go:122] "RemoveContainer" containerID="77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe" Nov 03 23:06:42.018494 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:42.018474 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe\": container with ID starting with 77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe not found: ID does not exist" containerID="77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe" Nov 03 23:06:42.018546 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.018502 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe"} err="failed to get container status \"77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe\": rpc error: code = NotFound desc = could not find container \"77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe\": container with ID starting with 77e1187f8feb73553b0b4115e5206bb6f8bb6994ed029e096bf49c721428c4fe not found: ID does not exist" Nov 03 23:06:42.018546 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.018519 4903 scope.go:122] "RemoveContainer" containerID="96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd" Nov 03 23:06:42.027967 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.027952 4903 scope.go:122] "RemoveContainer" containerID="96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd" Nov 03 23:06:42.028229 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:42.028210 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd\": container with ID starting with 96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd not found: ID does not exist" containerID="96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd" Nov 03 23:06:42.028275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.028236 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd"} err="failed to get container status \"96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd\": rpc error: code = NotFound desc = could not find container \"96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd\": container with ID starting with 96ae21eb22cf1149db197110d9691ab80a8bec3ffa1574de6af7a43f32d654bd not found: ID does not exist" Nov 03 23:06:42.028275 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.028252 4903 scope.go:122] "RemoveContainer" containerID="8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2" Nov 03 23:06:42.037206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.037187 4903 scope.go:122] "RemoveContainer" containerID="8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2" Nov 03 23:06:42.037507 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:42.037486 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2\": container with ID starting with 8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2 not found: ID does not exist" containerID="8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2" Nov 03 23:06:42.037558 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.037514 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2"} err="failed to get container status \"8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2\": rpc error: code = NotFound desc = could not find container \"8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2\": container with ID starting with 8035851c30f862ecc3c097b8914b2ffdc6b051ee6a263fd2749503c39fe104d2 not found: ID does not exist" Nov 03 23:06:42.037558 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.037531 4903 scope.go:122] "RemoveContainer" containerID="f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3" Nov 03 23:06:42.046963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.046943 4903 scope.go:122] "RemoveContainer" containerID="f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3" Nov 03 23:06:42.047216 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:42.047195 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3\": container with ID starting with f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3 not found: ID does not exist" containerID="f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3" Nov 03 23:06:42.047287 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.047226 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3"} err="failed to get container status \"f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3\": rpc error: code = NotFound desc = could not find container \"f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3\": container with ID starting with f03d872efebda9c232f1b1a270e3e2fac25c4ad2c53c82d4edb80c36778085b3 not found: ID does not exist" Nov 03 23:06:42.047287 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.047250 4903 scope.go:122] "RemoveContainer" containerID="a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a" Nov 03 23:06:42.055177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.055153 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb"] Nov 03 23:06:42.057571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.057549 4903 scope.go:122] "RemoveContainer" containerID="a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a" Nov 03 23:06:42.057908 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:42.057887 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a\": container with ID starting with a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a not found: ID does not exist" containerID="a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a" Nov 03 23:06:42.057984 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.057915 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a"} err="failed to get container status \"a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a\": rpc error: code = NotFound desc = could not find container \"a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a\": container with ID starting with a7b0bbf183f05280f5ec8881886e8712e02cff15eda1d5ae034812af9ec6622a not found: ID does not exist" Nov 03 23:06:42.059629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.059611 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/premium-w6-2d619f-kserve-75c689cbb8-zlsbb"] Nov 03 23:06:42.074623 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.074601 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d08ad03-a13d-4b82-a25d-00f821e29725-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:42.074623 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.074624 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpwhq\" (UniqueName: \"kubernetes.io/projected/7d08ad03-a13d-4b82-a25d-00f821e29725-kube-api-access-rpwhq\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:42.074772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.074632 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d08ad03-a13d-4b82-a25d-00f821e29725-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:42.090565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.090537 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj"] Nov 03 23:06:42.096816 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.096794 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d8a3d1/rate-test-model-d8a3d1-kserve-6f759ddfc7-f9lvj"] Nov 03 23:06:42.128312 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.128260 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv"] Nov 03 23:06:42.134838 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.134812 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a940d5-5b5d7f6f59-nvgzv"] Nov 03 23:06:42.169345 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.169320 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs"] Nov 03 23:06:42.172015 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.171992 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2d619f/simulator-w6-2d619f-kserve-7bbbcc87df-v6vvs"] Nov 03 23:06:42.201643 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.201610 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz"] Nov 03 23:06:42.204388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:42.204365 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a940d5-64d8f95f5b-l6fqz"] Nov 03 23:06:43.732752 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:43.732716 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c106a89-020e-4116-9077-19f6304b4685" path="/var/lib/kubelet/pods/1c106a89-020e-4116-9077-19f6304b4685/volumes" Nov 03 23:06:43.733263 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:43.733149 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" path="/var/lib/kubelet/pods/599f7b09-4f28-499c-bb80-cf9811e68a73/volumes" Nov 03 23:06:43.733555 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:43.733541 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d08ad03-a13d-4b82-a25d-00f821e29725" path="/var/lib/kubelet/pods/7d08ad03-a13d-4b82-a25d-00f821e29725/volumes" Nov 03 23:06:43.734224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:43.734208 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" path="/var/lib/kubelet/pods/806bbbeb-b428-44e2-95c1-4cf3282eb1cd/volumes" Nov 03 23:06:43.734620 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:43.734606 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3f5424-1ea1-4178-88e3-64b8f00bf1fd" path="/var/lib/kubelet/pods/eb3f5424-1ea1-4178-88e3-64b8f00bf1fd/volumes" Nov 03 23:06:44.767322 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:44.767254 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z"] Nov 03 23:06:44.773610 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:44.773574 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-a940d5-7wk8z"] Nov 03 23:06:45.733167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:45.733133 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad41311-0910-4d7f-980f-9c4062aa48c1" path="/var/lib/kubelet/pods/aad41311-0910-4d7f-980f-9c4062aa48c1/volumes" Nov 03 23:06:48.000307 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.000263 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk"] Nov 03 23:06:48.000819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.000586 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" containerID="cri-o://a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46" gracePeriod=30 Nov 03 23:06:48.383442 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:48.383355 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:06:48.557911 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.557875 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd"] Nov 03 23:06:48.558466 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558446 4903 cpu_manager.go:414] "containerMap: removing container" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" Nov 03 23:06:48.558536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558468 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" Nov 03 23:06:48.558536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558480 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1c106a89-020e-4116-9077-19f6304b4685" containerName="payload-processing" Nov 03 23:06:48.558536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558488 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c106a89-020e-4116-9077-19f6304b4685" containerName="payload-processing" Nov 03 23:06:48.558536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558504 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5f387f8d-1889-4269-b3b2-29fe3ba25667" containerName="payload-pre-processing" Nov 03 23:06:48.558536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558512 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f387f8d-1889-4269-b3b2-29fe3ba25667" containerName="payload-pre-processing" Nov 03 23:06:48.558536 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558533 4903 cpu_manager.go:414] "containerMap: removing container" podUID="07c4c43b-3b7e-4b8b-8a0f-b813d655c987" containerName="revoke-keys" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558540 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c4c43b-3b7e-4b8b-8a0f-b813d655c987" containerName="revoke-keys" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558556 4903 cpu_manager.go:414] "containerMap: removing container" podUID="65e10c32-f03d-4b47-9b73-c04087642e9b" containerName="revoke-keys" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558565 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e10c32-f03d-4b47-9b73-c04087642e9b" containerName="revoke-keys" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558580 4903 cpu_manager.go:414] "containerMap: removing container" podUID="08a1b497-df74-4ee0-aa16-447d4a1f4c06" containerName="main" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558587 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a1b497-df74-4ee0-aa16-447d4a1f4c06" containerName="main" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558610 4903 cpu_manager.go:414] "containerMap: removing container" podUID="eb3f5424-1ea1-4178-88e3-64b8f00bf1fd" containerName="payload-pre-processing" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558618 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3f5424-1ea1-4178-88e3-64b8f00bf1fd" containerName="payload-pre-processing" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558629 4903 cpu_manager.go:414] "containerMap: removing container" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558636 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558655 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7d08ad03-a13d-4b82-a25d-00f821e29725" containerName="main" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558662 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d08ad03-a13d-4b82-a25d-00f821e29725" containerName="main" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558673 4903 cpu_manager.go:414] "containerMap: removing container" podUID="76294da7-a114-4504-8530-a7bb4b3136be" containerName="payload-processing" Nov 03 23:06:48.558743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558680 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76294da7-a114-4504-8530-a7bb4b3136be" containerName="payload-processing" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558792 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e10c32-f03d-4b47-9b73-c04087642e9b" containerName="revoke-keys" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558806 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a1b497-df74-4ee0-aa16-447d4a1f4c06" containerName="main" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558818 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3f5424-1ea1-4178-88e3-64b8f00bf1fd" containerName="payload-pre-processing" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558828 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="599f7b09-4f28-499c-bb80-cf9811e68a73" containerName="main" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558838 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d08ad03-a13d-4b82-a25d-00f821e29725" containerName="main" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558845 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="76294da7-a114-4504-8530-a7bb4b3136be" containerName="payload-processing" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558857 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="806bbbeb-b428-44e2-95c1-4cf3282eb1cd" containerName="main" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558864 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c106a89-020e-4116-9077-19f6304b4685" containerName="payload-processing" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558873 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f387f8d-1889-4269-b3b2-29fe3ba25667" containerName="payload-pre-processing" Nov 03 23:06:48.559156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.558883 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c4c43b-3b7e-4b8b-8a0f-b813d655c987" containerName="revoke-keys" Nov 03 23:06:48.564159 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.564139 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.577579 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.577555 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:48.577704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.577664 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:48.579269 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.579248 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd"] Nov 03 23:06:48.617836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.617802 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:06:48.618013 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.617848 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:48.644534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.644464 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.644672 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.644651 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv22l\" (UniqueName: \"kubernetes.io/projected/596e2fe3-d80e-4803-9011-7eb2b840fc59-kube-api-access-pv22l\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.644722 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.644699 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.746071 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.746035 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv22l\" (UniqueName: \"kubernetes.io/projected/596e2fe3-d80e-4803-9011-7eb2b840fc59-kube-api-access-pv22l\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.746239 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.746087 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.746239 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.746128 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.746985 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.746960 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.748483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.748464 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.771156 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.771128 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv22l\" (UniqueName: \"kubernetes.io/projected/596e2fe3-d80e-4803-9011-7eb2b840fc59-kube-api-access-pv22l\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:48.877893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:48.877859 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:49.189905 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:49.189878 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd"] Nov 03 23:06:49.191653 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:49.191628 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596e2fe3_d80e_4803_9011_7eb2b840fc59.slice/crio-449ec8c5c8815b3b598905ff6e928ce578eb120976499a4765b629890ecc0d2b WatchSource:0}: Error finding container 449ec8c5c8815b3b598905ff6e928ce578eb120976499a4765b629890ecc0d2b: Status 404 returned error can't find the container with id 449ec8c5c8815b3b598905ff6e928ce578eb120976499a4765b629890ecc0d2b Nov 03 23:06:50.083999 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:50.083963 4903 generic.go:356] "Generic (PLEG): container finished" podID="596e2fe3-d80e-4803-9011-7eb2b840fc59" containerID="06b80075c4bd7a60ef7ea6e4707649a6ea3065df5cc468c1c3ad27a5da46b576" exitCode=0 Nov 03 23:06:50.084168 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:50.084045 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" event={"ID":"596e2fe3-d80e-4803-9011-7eb2b840fc59","Type":"ContainerDied","Data":"06b80075c4bd7a60ef7ea6e4707649a6ea3065df5cc468c1c3ad27a5da46b576"} Nov 03 23:06:50.084168 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:50.084129 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" event={"ID":"596e2fe3-d80e-4803-9011-7eb2b840fc59","Type":"ContainerStarted","Data":"449ec8c5c8815b3b598905ff6e928ce578eb120976499a4765b629890ecc0d2b"} Nov 03 23:06:51.029013 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.028972 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg"] Nov 03 23:06:51.039060 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.039017 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r"] Nov 03 23:06:51.041520 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.041496 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b2363a3-sksbg"] Nov 03 23:06:51.043667 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.043644 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82082c7d-s5q6r"] Nov 03 23:06:51.236672 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.236648 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:51.375211 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.375122 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596e2fe3-d80e-4803-9011-7eb2b840fc59-kube-api-access-pv22l\" (UniqueName: \"kubernetes.io/projected/596e2fe3-d80e-4803-9011-7eb2b840fc59-kube-api-access-pv22l\") pod \"596e2fe3-d80e-4803-9011-7eb2b840fc59\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " Nov 03 23:06:51.375211 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.375187 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-openshift-ca\") pod \"596e2fe3-d80e-4803-9011-7eb2b840fc59\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " Nov 03 23:06:51.375565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.375336 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-xks-ca\") pod \"596e2fe3-d80e-4803-9011-7eb2b840fc59\" (UID: \"596e2fe3-d80e-4803-9011-7eb2b840fc59\") " Nov 03 23:06:51.375666 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.375639 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-openshift-ca" pod "596e2fe3-d80e-4803-9011-7eb2b840fc59" (UID: "596e2fe3-d80e-4803-9011-7eb2b840fc59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:51.376038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.376012 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:51.379452 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.379404 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596e2fe3-d80e-4803-9011-7eb2b840fc59-kube-api-access-pv22l" pod "596e2fe3-d80e-4803-9011-7eb2b840fc59" (UID: "596e2fe3-d80e-4803-9011-7eb2b840fc59"). InnerVolumeSpecName "kube-api-access-pv22l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:51.379990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.379966 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-xks-ca" pod "596e2fe3-d80e-4803-9011-7eb2b840fc59" (UID: "596e2fe3-d80e-4803-9011-7eb2b840fc59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:51.477349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.477319 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv22l\" (UniqueName: \"kubernetes.io/projected/596e2fe3-d80e-4803-9011-7eb2b840fc59-kube-api-access-pv22l\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:51.477349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.477347 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/596e2fe3-d80e-4803-9011-7eb2b840fc59-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:51.732728 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.732688 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d3d808-44b5-42da-8cf6-14e48559be90" path="/var/lib/kubelet/pods/11d3d808-44b5-42da-8cf6-14e48559be90/volumes" Nov 03 23:06:51.733274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:51.733251 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca80af2-fc99-4d77-9c90-7427dda82697" path="/var/lib/kubelet/pods/fca80af2-fc99-4d77-9c90-7427dda82697/volumes" Nov 03 23:06:52.101120 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:52.101037 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" event={"ID":"596e2fe3-d80e-4803-9011-7eb2b840fc59","Type":"ContainerDied","Data":"449ec8c5c8815b3b598905ff6e928ce578eb120976499a4765b629890ecc0d2b"} Nov 03 23:06:52.101120 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:52.101083 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd" Nov 03 23:06:52.101557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:52.101087 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="449ec8c5c8815b3b598905ff6e928ce578eb120976499a4765b629890ecc0d2b" Nov 03 23:06:52.136119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:52.136082 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt"] Nov 03 23:06:52.138608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:52.138580 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-2d619f-86kkt"] Nov 03 23:06:53.285097 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.285037 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 23:06:53.399120 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399084 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-credential-socket\") pod \"f9847228-3a90-4625-a6b0-e17b44bed10e\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " Nov 03 23:06:53.399340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399141 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9847228-3a90-4625-a6b0-e17b44bed10e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9847228-3a90-4625-a6b0-e17b44bed10e-istiod-ca-cert\") pod \"f9847228-3a90-4625-a6b0-e17b44bed10e\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " Nov 03 23:06:53.399340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399171 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-token\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-token\") pod \"f9847228-3a90-4625-a6b0-e17b44bed10e\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " Nov 03 23:06:53.399340 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399202 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-socket\") pod \"f9847228-3a90-4625-a6b0-e17b44bed10e\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " Nov 03 23:06:53.399544 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399452 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-credential-socket" pod "f9847228-3a90-4625-a6b0-e17b44bed10e" (UID: "f9847228-3a90-4625-a6b0-e17b44bed10e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:53.399544 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399472 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-envoy\") pod \"f9847228-3a90-4625-a6b0-e17b44bed10e\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " Nov 03 23:06:53.399657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399552 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-socket" pod "f9847228-3a90-4625-a6b0-e17b44bed10e" (UID: "f9847228-3a90-4625-a6b0-e17b44bed10e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:53.399657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399560 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-data\") pod \"f9847228-3a90-4625-a6b0-e17b44bed10e\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " Nov 03 23:06:53.399657 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399614 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-certs\") pod \"f9847228-3a90-4625-a6b0-e17b44bed10e\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " Nov 03 23:06:53.399816 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399654 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-podinfo\") pod \"f9847228-3a90-4625-a6b0-e17b44bed10e\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " Nov 03 23:06:53.399816 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399613 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9847228-3a90-4625-a6b0-e17b44bed10e-istiod-ca-cert" pod "f9847228-3a90-4625-a6b0-e17b44bed10e" (UID: "f9847228-3a90-4625-a6b0-e17b44bed10e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:53.399816 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399706 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-kube-api-access-pmzkl\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-kube-api-access-pmzkl\") pod \"f9847228-3a90-4625-a6b0-e17b44bed10e\" (UID: \"f9847228-3a90-4625-a6b0-e17b44bed10e\") " Nov 03 23:06:53.399984 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399808 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-data" pod "f9847228-3a90-4625-a6b0-e17b44bed10e" (UID: "f9847228-3a90-4625-a6b0-e17b44bed10e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:53.399984 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.399875 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-certs" pod "f9847228-3a90-4625-a6b0-e17b44bed10e" (UID: "f9847228-3a90-4625-a6b0-e17b44bed10e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:53.400168 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.400148 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9847228-3a90-4625-a6b0-e17b44bed10e-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:53.400246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.400172 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:53.400246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.400186 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:53.400246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.400199 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:53.400246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.400211 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:53.401897 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.401870 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-kube-api-access-pmzkl" pod "f9847228-3a90-4625-a6b0-e17b44bed10e" (UID: "f9847228-3a90-4625-a6b0-e17b44bed10e"). InnerVolumeSpecName "kube-api-access-pmzkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:53.401999 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.401924 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-envoy" pod "f9847228-3a90-4625-a6b0-e17b44bed10e" (UID: "f9847228-3a90-4625-a6b0-e17b44bed10e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:06:53.402057 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.402034 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-token" pod "f9847228-3a90-4625-a6b0-e17b44bed10e" (UID: "f9847228-3a90-4625-a6b0-e17b44bed10e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:53.402118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.402073 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-podinfo" pod "f9847228-3a90-4625-a6b0-e17b44bed10e" (UID: "f9847228-3a90-4625-a6b0-e17b44bed10e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:06:53.501431 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.501395 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:53.501431 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.501424 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:53.501431 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.501434 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmzkl\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-kube-api-access-pmzkl\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:53.501680 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.501445 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9847228-3a90-4625-a6b0-e17b44bed10e-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:53.735158 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:53.735124 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6567522c-adb2-4fe7-847b-8a330647ba9c" path="/var/lib/kubelet/pods/6567522c-adb2-4fe7-847b-8a330647ba9c/volumes" Nov 03 23:06:54.121974 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.121891 4903 generic.go:356] "Generic (PLEG): container finished" podID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerID="a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46" exitCode=0 Nov 03 23:06:54.121974 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.121961 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" event={"ID":"f9847228-3a90-4625-a6b0-e17b44bed10e","Type":"ContainerDied","Data":"a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46"} Nov 03 23:06:54.122157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.121988 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" Nov 03 23:06:54.122157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.122027 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk" event={"ID":"f9847228-3a90-4625-a6b0-e17b44bed10e","Type":"ContainerDied","Data":"7eba6bddb98b0daf40ae0980c15c69a4d6e8923453dc4d44959d7d8859346150"} Nov 03 23:06:54.122157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.122048 4903 scope.go:122] "RemoveContainer" containerID="a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46" Nov 03 23:06:54.132827 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.132803 4903 scope.go:122] "RemoveContainer" containerID="997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d" Nov 03 23:06:54.142606 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.142582 4903 scope.go:122] "RemoveContainer" containerID="a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46" Nov 03 23:06:54.142893 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:54.142874 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46\": container with ID starting with a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46 not found: ID does not exist" containerID="a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46" Nov 03 23:06:54.142970 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.142907 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46"} err="failed to get container status \"a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46\": rpc error: code = NotFound desc = could not find container \"a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46\": container with ID starting with a9594f0ad763611e7982aaaa2486554fc9bf13167bc96236033d3902ae3a9e46 not found: ID does not exist" Nov 03 23:06:54.142970 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.142932 4903 scope.go:122] "RemoveContainer" containerID="997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d" Nov 03 23:06:54.143203 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:06:54.143177 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d\": container with ID starting with 997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d not found: ID does not exist" containerID="997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d" Nov 03 23:06:54.143252 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.143208 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d"} err="failed to get container status \"997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d\": rpc error: code = NotFound desc = could not find container \"997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d\": container with ID starting with 997aff6e1291c9ba33f5f65a794d6ad6c354a80728de36a4a5965b8afbf5545d not found: ID does not exist" Nov 03 23:06:54.169016 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.168987 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk"] Nov 03 23:06:54.176059 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.176028 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-23d281-openshift-default-5cb785989-8jstk"] Nov 03 23:06:54.564879 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.564842 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx"] Nov 03 23:06:54.565304 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.565280 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" Nov 03 23:06:54.565349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.565307 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" Nov 03 23:06:54.565349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.565319 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" Nov 03 23:06:54.565349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.565325 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" Nov 03 23:06:54.565349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.565338 4903 cpu_manager.go:414] "containerMap: removing container" podUID="596e2fe3-d80e-4803-9011-7eb2b840fc59" containerName="revoke-keys" Nov 03 23:06:54.565349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.565344 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596e2fe3-d80e-4803-9011-7eb2b840fc59" containerName="revoke-keys" Nov 03 23:06:54.565517 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.565490 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" Nov 03 23:06:54.565517 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.565502 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" containerName="istio-proxy" Nov 03 23:06:54.565517 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.565514 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="596e2fe3-d80e-4803-9011-7eb2b840fc59" containerName="revoke-keys" Nov 03 23:06:54.570176 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.570159 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.587946 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.587922 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx"] Nov 03 23:06:54.595463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.595425 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:06:54.595615 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.595597 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:06:54.595694 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.595666 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:06:54.595793 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.595772 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:06:54.715872 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.715829 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.716173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.716132 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws45n\" (UniqueName: \"kubernetes.io/projected/4a72a922-9d79-4e13-a951-13cf1a99abe1-kube-api-access-ws45n\") pod \"maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.716343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.716212 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.817950 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.817869 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.817950 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.817923 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws45n\" (UniqueName: \"kubernetes.io/projected/4a72a922-9d79-4e13-a951-13cf1a99abe1-kube-api-access-ws45n\") pod \"maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.817950 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.817948 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.818683 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.818661 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.820277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.820256 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.839760 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.839731 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws45n\" (UniqueName: \"kubernetes.io/projected/4a72a922-9d79-4e13-a951-13cf1a99abe1-kube-api-access-ws45n\") pod \"maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:54.884125 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:54.884094 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:55.187565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:55.187539 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx"] Nov 03 23:06:55.188915 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:06:55.188888 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a72a922_9d79_4e13_a951_13cf1a99abe1.slice/crio-d17ffc72330bfb8b61b9d505232cc91dd0fe3efe7c1c2c46e252781261f99725 WatchSource:0}: Error finding container d17ffc72330bfb8b61b9d505232cc91dd0fe3efe7c1c2c46e252781261f99725: Status 404 returned error can't find the container with id d17ffc72330bfb8b61b9d505232cc91dd0fe3efe7c1c2c46e252781261f99725 Nov 03 23:06:55.732972 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:55.732900 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9847228-3a90-4625-a6b0-e17b44bed10e" path="/var/lib/kubelet/pods/f9847228-3a90-4625-a6b0-e17b44bed10e/volumes" Nov 03 23:06:56.152313 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:56.152158 4903 generic.go:356] "Generic (PLEG): container finished" podID="4a72a922-9d79-4e13-a951-13cf1a99abe1" containerID="a2ea66dbfdd72868dcf9767709d38b253595f06d4c3033a3206c4cc9beaaee6c" exitCode=0 Nov 03 23:06:56.152313 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:56.152255 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" event={"ID":"4a72a922-9d79-4e13-a951-13cf1a99abe1","Type":"ContainerDied","Data":"a2ea66dbfdd72868dcf9767709d38b253595f06d4c3033a3206c4cc9beaaee6c"} Nov 03 23:06:56.152577 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:56.152354 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" event={"ID":"4a72a922-9d79-4e13-a951-13cf1a99abe1","Type":"ContainerStarted","Data":"d17ffc72330bfb8b61b9d505232cc91dd0fe3efe7c1c2c46e252781261f99725"} Nov 03 23:06:57.329018 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.328988 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:57.446365 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.446260 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a72a922-9d79-4e13-a951-13cf1a99abe1-kube-api-access-ws45n\" (UniqueName: \"kubernetes.io/projected/4a72a922-9d79-4e13-a951-13cf1a99abe1-kube-api-access-ws45n\") pod \"4a72a922-9d79-4e13-a951-13cf1a99abe1\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " Nov 03 23:06:57.446540 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.446421 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-xks-ca\") pod \"4a72a922-9d79-4e13-a951-13cf1a99abe1\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " Nov 03 23:06:57.446540 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.446454 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-openshift-ca\") pod \"4a72a922-9d79-4e13-a951-13cf1a99abe1\" (UID: \"4a72a922-9d79-4e13-a951-13cf1a99abe1\") " Nov 03 23:06:57.446874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.446849 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-openshift-ca" pod "4a72a922-9d79-4e13-a951-13cf1a99abe1" (UID: "4a72a922-9d79-4e13-a951-13cf1a99abe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:06:57.448571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.448548 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a72a922-9d79-4e13-a951-13cf1a99abe1-kube-api-access-ws45n" pod "4a72a922-9d79-4e13-a951-13cf1a99abe1" (UID: "4a72a922-9d79-4e13-a951-13cf1a99abe1"). InnerVolumeSpecName "kube-api-access-ws45n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:06:57.448686 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.448585 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-xks-ca" pod "4a72a922-9d79-4e13-a951-13cf1a99abe1" (UID: "4a72a922-9d79-4e13-a951-13cf1a99abe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:06:57.547848 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.547808 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws45n\" (UniqueName: \"kubernetes.io/projected/4a72a922-9d79-4e13-a951-13cf1a99abe1-kube-api-access-ws45n\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:57.547848 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.547849 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:57.548078 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:57.547866 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a72a922-9d79-4e13-a951-13cf1a99abe1-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:06:58.168363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:58.168334 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" Nov 03 23:06:58.168535 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:58.168355 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx" event={"ID":"4a72a922-9d79-4e13-a951-13cf1a99abe1","Type":"ContainerDied","Data":"d17ffc72330bfb8b61b9d505232cc91dd0fe3efe7c1c2c46e252781261f99725"} Nov 03 23:06:58.168535 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:58.168409 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d17ffc72330bfb8b61b9d505232cc91dd0fe3efe7c1c2c46e252781261f99725" Nov 03 23:06:59.004252 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:59.004215 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t"] Nov 03 23:06:59.340808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:06:59.340725 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg"] Nov 03 23:07:00.371574 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:00.371536 4903 scope.go:122] "RemoveContainer" containerID="36334ffdf4ba75fb3f12938064e8b014923965573a4f437bd1b67ebe12c79747" Nov 03 23:07:00.381885 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:00.381862 4903 scope.go:122] "RemoveContainer" containerID="848945c80b46a9e579c88fdd66bb1ef7b66c0115ae6cb80ae5f3522aa3f017fd" Nov 03 23:07:00.391492 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:00.391472 4903 scope.go:122] "RemoveContainer" containerID="35cf692a37c8053d6a081a0667ad79fd1e2ec3c30cd3920d5ea00c016b55b149" Nov 03 23:07:00.400645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:00.400624 4903 scope.go:122] "RemoveContainer" containerID="dbc7d996452f654fc0bab1992f85ca39a6df2e6f867e22c39c36ac46f1d50e02" Nov 03 23:07:04.004945 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:04.004892 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" podUID="fddc0ee5-3581-450e-b82a-ca28c12cd15e" containerName="payload-pre-processing" containerID="cri-o://5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30" gracePeriod=25 Nov 03 23:07:04.590181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:04.590140 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd"] Nov 03 23:07:04.599555 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:04.599523 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d8a3d1-9gncd"] Nov 03 23:07:05.076080 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:05.076044 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn"] Nov 03 23:07:05.733553 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:05.733517 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596e2fe3-d80e-4803-9011-7eb2b840fc59" path="/var/lib/kubelet/pods/596e2fe3-d80e-4803-9011-7eb2b840fc59/volumes" Nov 03 23:07:07.884750 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.884708 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj"] Nov 03 23:07:07.885364 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.885340 4903 cpu_manager.go:414] "containerMap: removing container" podUID="4a72a922-9d79-4e13-a951-13cf1a99abe1" containerName="revoke-keys" Nov 03 23:07:07.885556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.885538 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a72a922-9d79-4e13-a951-13cf1a99abe1" containerName="revoke-keys" Nov 03 23:07:07.885839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.885816 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a72a922-9d79-4e13-a951-13cf1a99abe1" containerName="revoke-keys" Nov 03 23:07:07.892512 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.892491 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:07.907099 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.907074 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj"] Nov 03 23:07:07.913899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.913605 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:07:07.913899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.913680 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:07:07.913899 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.913709 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:07.920470 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.920446 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:07:07.965736 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.965702 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgbgf\" (UniqueName: \"kubernetes.io/projected/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-kube-api-access-lgbgf\") pod \"maas-api-revoke-sub-d6db3168-67xzj\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:07.965922 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.965746 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6db3168-67xzj\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:07.965922 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:07.965832 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6db3168-67xzj\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:08.066784 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:08.066748 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbgf\" (UniqueName: \"kubernetes.io/projected/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-kube-api-access-lgbgf\") pod \"maas-api-revoke-sub-d6db3168-67xzj\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:08.066993 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:08.066790 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6db3168-67xzj\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:08.066993 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:08.066818 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6db3168-67xzj\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:08.067470 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:08.067441 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6db3168-67xzj\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:08.069158 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:08.069139 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6db3168-67xzj\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:08.088562 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:08.088537 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbgf\" (UniqueName: \"kubernetes.io/projected/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-kube-api-access-lgbgf\") pod \"maas-api-revoke-sub-d6db3168-67xzj\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:08.207008 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:08.206973 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:08.526351 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:08.526318 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj"] Nov 03 23:07:08.528315 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:07:08.528268 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda101b815_7b9a_4f8a_9d5c_31b3c4c4defd.slice/crio-7b8d8614a79f224336b61e4c1bd477cd544e309d8e83600567aec1e3fd875fda WatchSource:0}: Error finding container 7b8d8614a79f224336b61e4c1bd477cd544e309d8e83600567aec1e3fd875fda: Status 404 returned error can't find the container with id 7b8d8614a79f224336b61e4c1bd477cd544e309d8e83600567aec1e3fd875fda Nov 03 23:07:09.258809 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:09.258775 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" event={"ID":"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd","Type":"ContainerStarted","Data":"bba5697ed3aa8e78ea422f40add58016e9998686101e46bdea44fedf17029cc8"} Nov 03 23:07:09.259332 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:09.258832 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" event={"ID":"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd","Type":"ContainerStarted","Data":"7b8d8614a79f224336b61e4c1bd477cd544e309d8e83600567aec1e3fd875fda"} Nov 03 23:07:09.300726 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:09.300669 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" podStartSLOduration=2.3006555410000002 podStartE2EDuration="2.300655541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:09.297312589 +0000 UTC m=+3255.701253561" watchObservedRunningTime="2026-11-03 23:07:09.300655541 +0000 UTC m=+3255.704596516" Nov 03 23:07:10.077174 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.077130 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" podUID="f255b355-f63f-494e-96df-40136203d990" containerName="payload-pre-processing" containerID="cri-o://6786fb16605e0d6f78d83480e6445d8da693e7f7c215703a1334cf29de55480e" gracePeriod=25 Nov 03 23:07:10.269324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.269260 4903 generic.go:356] "Generic (PLEG): container finished" podID="a101b815-7b9a-4f8a-9d5c-31b3c4c4defd" containerID="bba5697ed3aa8e78ea422f40add58016e9998686101e46bdea44fedf17029cc8" exitCode=0 Nov 03 23:07:10.269799 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.269442 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" event={"ID":"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd","Type":"ContainerDied","Data":"bba5697ed3aa8e78ea422f40add58016e9998686101e46bdea44fedf17029cc8"} Nov 03 23:07:10.271409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.271377 4903 generic.go:356] "Generic (PLEG): container finished" podID="f255b355-f63f-494e-96df-40136203d990" containerID="6786fb16605e0d6f78d83480e6445d8da693e7f7c215703a1334cf29de55480e" exitCode=0 Nov 03 23:07:10.271517 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.271450 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" event={"ID":"f255b355-f63f-494e-96df-40136203d990","Type":"ContainerDied","Data":"6786fb16605e0d6f78d83480e6445d8da693e7f7c215703a1334cf29de55480e"} Nov 03 23:07:10.340004 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.339980 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 23:07:10.390366 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.390333 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f255b355-f63f-494e-96df-40136203d990-kube-api-access-l8l5b\" (UniqueName: \"kubernetes.io/projected/f255b355-f63f-494e-96df-40136203d990-kube-api-access-l8l5b\") pod \"f255b355-f63f-494e-96df-40136203d990\" (UID: \"f255b355-f63f-494e-96df-40136203d990\") " Nov 03 23:07:10.390570 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.390398 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f255b355-f63f-494e-96df-40136203d990-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f255b355-f63f-494e-96df-40136203d990-plugins-config-volume\") pod \"f255b355-f63f-494e-96df-40136203d990\" (UID: \"f255b355-f63f-494e-96df-40136203d990\") " Nov 03 23:07:10.390838 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.390804 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f255b355-f63f-494e-96df-40136203d990-plugins-config-volume" pod "f255b355-f63f-494e-96df-40136203d990" (UID: "f255b355-f63f-494e-96df-40136203d990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:10.390977 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.390902 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f255b355-f63f-494e-96df-40136203d990-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:10.393030 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.393001 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f255b355-f63f-494e-96df-40136203d990-kube-api-access-l8l5b" pod "f255b355-f63f-494e-96df-40136203d990" (UID: "f255b355-f63f-494e-96df-40136203d990"). InnerVolumeSpecName "kube-api-access-l8l5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:10.491847 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.491811 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8l5b\" (UniqueName: \"kubernetes.io/projected/f255b355-f63f-494e-96df-40136203d990-kube-api-access-l8l5b\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:10.959848 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:10.959799 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" podUID="fddc0ee5-3581-450e-b82a-ca28c12cd15e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:07:11.280963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.280885 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" Nov 03 23:07:11.280963 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.280908 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn" event={"ID":"f255b355-f63f-494e-96df-40136203d990","Type":"ContainerDied","Data":"56ed9440e22fe183203b18e90a505fa48bce8a6faa1ded3a0c76efee3fff5d17"} Nov 03 23:07:11.281507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.280973 4903 scope.go:122] "RemoveContainer" containerID="6786fb16605e0d6f78d83480e6445d8da693e7f7c215703a1334cf29de55480e" Nov 03 23:07:11.337811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.337784 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn"] Nov 03 23:07:11.348854 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.348821 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-88a4d9-797cb8d49f-mfqtn"] Nov 03 23:07:11.430417 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.430385 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:11.502444 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.502414 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-kube-api-access-lgbgf\" (UniqueName: \"kubernetes.io/projected/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-kube-api-access-lgbgf\") pod \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " Nov 03 23:07:11.502654 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.502453 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-xks-ca\") pod \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " Nov 03 23:07:11.502654 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.502579 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-openshift-ca\") pod \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\" (UID: \"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd\") " Nov 03 23:07:11.503048 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.503021 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-openshift-ca" pod "a101b815-7b9a-4f8a-9d5c-31b3c4c4defd" (UID: "a101b815-7b9a-4f8a-9d5c-31b3c4c4defd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:11.504687 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.504647 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-kube-api-access-lgbgf" pod "a101b815-7b9a-4f8a-9d5c-31b3c4c4defd" (UID: "a101b815-7b9a-4f8a-9d5c-31b3c4c4defd"). InnerVolumeSpecName "kube-api-access-lgbgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:11.504793 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.504681 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-xks-ca" pod "a101b815-7b9a-4f8a-9d5c-31b3c4c4defd" (UID: "a101b815-7b9a-4f8a-9d5c-31b3c4c4defd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:11.604446 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.604357 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgbgf\" (UniqueName: \"kubernetes.io/projected/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-kube-api-access-lgbgf\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:11.604446 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.604397 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:11.604446 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.604408 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:11.732555 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:11.732519 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f255b355-f63f-494e-96df-40136203d990" path="/var/lib/kubelet/pods/f255b355-f63f-494e-96df-40136203d990/volumes" Nov 03 23:07:12.288841 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:12.288803 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" Nov 03 23:07:12.289342 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:12.288836 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj" event={"ID":"a101b815-7b9a-4f8a-9d5c-31b3c4c4defd","Type":"ContainerDied","Data":"7b8d8614a79f224336b61e4c1bd477cd544e309d8e83600567aec1e3fd875fda"} Nov 03 23:07:12.289342 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:12.288897 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b8d8614a79f224336b61e4c1bd477cd544e309d8e83600567aec1e3fd875fda" Nov 03 23:07:12.434450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:12.434401 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:07:12.718002 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:12.717962 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb"] Nov 03 23:07:12.718264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:12.718217 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerName="istio-proxy" containerID="cri-o://cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89" gracePeriod=30 Nov 03 23:07:13.431262 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.431227 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm"] Nov 03 23:07:13.431961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.431938 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f255b355-f63f-494e-96df-40136203d990" containerName="payload-pre-processing" Nov 03 23:07:13.432075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.431963 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f255b355-f63f-494e-96df-40136203d990" containerName="payload-pre-processing" Nov 03 23:07:13.432075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.432001 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a101b815-7b9a-4f8a-9d5c-31b3c4c4defd" containerName="revoke-keys" Nov 03 23:07:13.432075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.432009 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a101b815-7b9a-4f8a-9d5c-31b3c4c4defd" containerName="revoke-keys" Nov 03 23:07:13.432256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.432119 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f255b355-f63f-494e-96df-40136203d990" containerName="payload-pre-processing" Nov 03 23:07:13.432256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.432139 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a101b815-7b9a-4f8a-9d5c-31b3c4c4defd" containerName="revoke-keys" Nov 03 23:07:13.437022 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.436997 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.453562 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.453529 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm"] Nov 03 23:07:13.460032 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.460002 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:07:13.460584 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.460014 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:07:13.496804 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.496767 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:13.496990 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.496942 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:07:13.525770 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.525733 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.525948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.525784 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.526017 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.525993 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kkbg\" (UniqueName: \"kubernetes.io/projected/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-kube-api-access-2kkbg\") pod \"maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.627611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.627570 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kkbg\" (UniqueName: \"kubernetes.io/projected/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-kube-api-access-2kkbg\") pod \"maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.627854 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.627656 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.627854 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.627683 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.628445 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.628419 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.630076 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.630050 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.650553 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.650528 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kkbg\" (UniqueName: \"kubernetes.io/projected/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-kube-api-access-2kkbg\") pod \"maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:13.752107 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:13.752010 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:14.061705 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.061674 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm"] Nov 03 23:07:14.064043 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:07:14.064013 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8620ffe7_06ca_4e7c_8a28_aa8da0726cc9.slice/crio-59394e9b45c54047070b46eacd8c43e4e2a190bf0ea0696aca2469168cb30b47 WatchSource:0}: Error finding container 59394e9b45c54047070b46eacd8c43e4e2a190bf0ea0696aca2469168cb30b47: Status 404 returned error can't find the container with id 59394e9b45c54047070b46eacd8c43e4e2a190bf0ea0696aca2469168cb30b47 Nov 03 23:07:14.305016 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.304980 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" event={"ID":"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9","Type":"ContainerStarted","Data":"59394e9b45c54047070b46eacd8c43e4e2a190bf0ea0696aca2469168cb30b47"} Nov 03 23:07:14.361480 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.361443 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" containerID="cri-o://779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b" gracePeriod=45 Nov 03 23:07:14.606423 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.606356 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 23:07:14.741183 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741141 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-tmp-dir\") pod \"c164f82e-0816-4e68-a013-aea385eb9b83\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " Nov 03 23:07:14.741400 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741205 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-home\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-home\") pod \"c164f82e-0816-4e68-a013-aea385eb9b83\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " Nov 03 23:07:14.741400 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741342 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-model-cache\") pod \"c164f82e-0816-4e68-a013-aea385eb9b83\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " Nov 03 23:07:14.741400 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741387 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c164f82e-0816-4e68-a013-aea385eb9b83-kube-api-access-rsxv9\" (UniqueName: \"kubernetes.io/projected/c164f82e-0816-4e68-a013-aea385eb9b83-kube-api-access-rsxv9\") pod \"c164f82e-0816-4e68-a013-aea385eb9b83\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " Nov 03 23:07:14.741598 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741427 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c164f82e-0816-4e68-a013-aea385eb9b83-tls-certs\" (UniqueName: \"kubernetes.io/secret/c164f82e-0816-4e68-a013-aea385eb9b83-tls-certs\") pod \"c164f82e-0816-4e68-a013-aea385eb9b83\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " Nov 03 23:07:14.741598 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741474 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-dshm\") pod \"c164f82e-0816-4e68-a013-aea385eb9b83\" (UID: \"c164f82e-0816-4e68-a013-aea385eb9b83\") " Nov 03 23:07:14.741598 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741565 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-tmp-dir" pod "c164f82e-0816-4e68-a013-aea385eb9b83" (UID: "c164f82e-0816-4e68-a013-aea385eb9b83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:14.742167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741793 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-model-cache" pod "c164f82e-0816-4e68-a013-aea385eb9b83" (UID: "c164f82e-0816-4e68-a013-aea385eb9b83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:14.742167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741909 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.742167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.741928 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.742167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.742144 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-home" pod "c164f82e-0816-4e68-a013-aea385eb9b83" (UID: "c164f82e-0816-4e68-a013-aea385eb9b83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:14.745937 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.745911 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c164f82e-0816-4e68-a013-aea385eb9b83-kube-api-access-rsxv9" pod "c164f82e-0816-4e68-a013-aea385eb9b83" (UID: "c164f82e-0816-4e68-a013-aea385eb9b83"). InnerVolumeSpecName "kube-api-access-rsxv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:14.747879 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.747848 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-dshm" pod "c164f82e-0816-4e68-a013-aea385eb9b83" (UID: "c164f82e-0816-4e68-a013-aea385eb9b83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:14.748035 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.748010 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c164f82e-0816-4e68-a013-aea385eb9b83-tls-certs" pod "c164f82e-0816-4e68-a013-aea385eb9b83" (UID: "c164f82e-0816-4e68-a013-aea385eb9b83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:14.842984 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.842954 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.842984 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.842982 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxv9\" (UniqueName: \"kubernetes.io/projected/c164f82e-0816-4e68-a013-aea385eb9b83-kube-api-access-rsxv9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.843192 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.843066 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c164f82e-0816-4e68-a013-aea385eb9b83-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:14.843192 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:14.843076 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c164f82e-0816-4e68-a013-aea385eb9b83-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:15.313774 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.313696 4903 generic.go:356] "Generic (PLEG): container finished" podID="8620ffe7-06ca-4e7c-8a28-aa8da0726cc9" containerID="e33640548b66bf87c4b433fad514717feb80757ca888bb9c6148ecc2350e77a1" exitCode=0 Nov 03 23:07:15.313937 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.313783 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" event={"ID":"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9","Type":"ContainerDied","Data":"e33640548b66bf87c4b433fad514717feb80757ca888bb9c6148ecc2350e77a1"} Nov 03 23:07:15.315090 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.315065 4903 generic.go:356] "Generic (PLEG): container finished" podID="c164f82e-0816-4e68-a013-aea385eb9b83" containerID="779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b" exitCode=0 Nov 03 23:07:15.315198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.315109 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" Nov 03 23:07:15.315198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.315147 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" event={"ID":"c164f82e-0816-4e68-a013-aea385eb9b83","Type":"ContainerDied","Data":"779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b"} Nov 03 23:07:15.315198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.315180 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg" event={"ID":"c164f82e-0816-4e68-a013-aea385eb9b83","Type":"ContainerDied","Data":"1cb319d21cda164dc5516df58fb5eabb2b4ea79fe08d8f69cf5d4403090cc186"} Nov 03 23:07:15.315394 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.315199 4903 scope.go:122] "RemoveContainer" containerID="779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b" Nov 03 23:07:15.326232 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.326213 4903 scope.go:122] "RemoveContainer" containerID="779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b" Nov 03 23:07:15.326628 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:07:15.326604 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b\": container with ID starting with 779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b not found: ID does not exist" containerID="779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b" Nov 03 23:07:15.326705 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.326640 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b"} err="failed to get container status \"779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b\": rpc error: code = NotFound desc = could not find container \"779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b\": container with ID starting with 779f40746e63343d076d774eaf1cdd0970b4a848fd802c0a0b6e0788d8f4060b not found: ID does not exist" Nov 03 23:07:15.386021 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.385985 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg"] Nov 03 23:07:15.392607 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.392571 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-88a4d9/distinct-2-w0-88a4d9-kserve-756c8dd9b4-ckstg"] Nov 03 23:07:15.732595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:15.732552 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" path="/var/lib/kubelet/pods/c164f82e-0816-4e68-a013-aea385eb9b83/volumes" Nov 03 23:07:16.468473 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.468446 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:16.561808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.561770 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-xks-ca\") pod \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " Nov 03 23:07:16.561985 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.561873 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-kube-api-access-2kkbg\" (UniqueName: \"kubernetes.io/projected/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-kube-api-access-2kkbg\") pod \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " Nov 03 23:07:16.562077 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.562022 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-openshift-ca\") pod \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\" (UID: \"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9\") " Nov 03 23:07:16.562423 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.562400 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-openshift-ca" pod "8620ffe7-06ca-4e7c-8a28-aa8da0726cc9" (UID: "8620ffe7-06ca-4e7c-8a28-aa8da0726cc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:16.563930 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.563903 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-kube-api-access-2kkbg" pod "8620ffe7-06ca-4e7c-8a28-aa8da0726cc9" (UID: "8620ffe7-06ca-4e7c-8a28-aa8da0726cc9"). InnerVolumeSpecName "kube-api-access-2kkbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:16.564037 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.563948 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-xks-ca" pod "8620ffe7-06ca-4e7c-8a28-aa8da0726cc9" (UID: "8620ffe7-06ca-4e7c-8a28-aa8da0726cc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:16.663121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.663082 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kkbg\" (UniqueName: \"kubernetes.io/projected/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-kube-api-access-2kkbg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:16.663121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.663117 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:16.663349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:16.663131 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:17.331058 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:17.331024 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" Nov 03 23:07:17.331508 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:17.331071 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm" event={"ID":"8620ffe7-06ca-4e7c-8a28-aa8da0726cc9","Type":"ContainerDied","Data":"59394e9b45c54047070b46eacd8c43e4e2a190bf0ea0696aca2469168cb30b47"} Nov 03 23:07:17.331508 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:17.331129 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59394e9b45c54047070b46eacd8c43e4e2a190bf0ea0696aca2469168cb30b47" Nov 03 23:07:17.981928 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:17.981897 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 23:07:18.076268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076178 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-kube-api-access-t5ngr\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-kube-api-access-t5ngr\") pod \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " Nov 03 23:07:18.076268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076239 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-credential-socket\") pod \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " Nov 03 23:07:18.076487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076303 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-socket\") pod \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " Nov 03 23:07:18.076487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076348 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-token\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-token\") pod \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " Nov 03 23:07:18.076487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076415 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-certs\") pod \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " Nov 03 23:07:18.076487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076464 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-data\") pod \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " Nov 03 23:07:18.076695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076497 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istiod-ca-cert\") pod \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " Nov 03 23:07:18.076695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076529 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-envoy\") pod \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " Nov 03 23:07:18.076695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076571 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-podinfo\") pod \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\" (UID: \"93642a17-3a9f-4321-ad02-4d9c26fa42c2\") " Nov 03 23:07:18.076695 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076576 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-credential-socket" pod "93642a17-3a9f-4321-ad02-4d9c26fa42c2" (UID: "93642a17-3a9f-4321-ad02-4d9c26fa42c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.076926 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076873 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-certs" pod "93642a17-3a9f-4321-ad02-4d9c26fa42c2" (UID: "93642a17-3a9f-4321-ad02-4d9c26fa42c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.076989 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.076959 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istiod-ca-cert" pod "93642a17-3a9f-4321-ad02-4d9c26fa42c2" (UID: "93642a17-3a9f-4321-ad02-4d9c26fa42c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:18.077060 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.077043 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.077112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.077065 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.077263 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.077239 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-socket" pod "93642a17-3a9f-4321-ad02-4d9c26fa42c2" (UID: "93642a17-3a9f-4321-ad02-4d9c26fa42c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.077406 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.077271 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-data" pod "93642a17-3a9f-4321-ad02-4d9c26fa42c2" (UID: "93642a17-3a9f-4321-ad02-4d9c26fa42c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.078884 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.078856 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-podinfo" pod "93642a17-3a9f-4321-ad02-4d9c26fa42c2" (UID: "93642a17-3a9f-4321-ad02-4d9c26fa42c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:18.079007 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.078963 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-kube-api-access-t5ngr" pod "93642a17-3a9f-4321-ad02-4d9c26fa42c2" (UID: "93642a17-3a9f-4321-ad02-4d9c26fa42c2"). InnerVolumeSpecName "kube-api-access-t5ngr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:18.079184 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.079160 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-token" pod "93642a17-3a9f-4321-ad02-4d9c26fa42c2" (UID: "93642a17-3a9f-4321-ad02-4d9c26fa42c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:18.079184 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.079176 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-envoy" pod "93642a17-3a9f-4321-ad02-4d9c26fa42c2" (UID: "93642a17-3a9f-4321-ad02-4d9c26fa42c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:18.174507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.174467 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk"] Nov 03 23:07:18.178634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.178602 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.178634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.178633 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.178798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.178645 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5ngr\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-kube-api-access-t5ngr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.178798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.178656 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.178798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.178664 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.178798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.178672 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.178798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.178680 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93642a17-3a9f-4321-ad02-4d9c26fa42c2-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:18.203099 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.203073 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5"] Nov 03 23:07:18.339207 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.339116 4903 generic.go:356] "Generic (PLEG): container finished" podID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerID="cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89" exitCode=0 Nov 03 23:07:18.339207 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.339170 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" Nov 03 23:07:18.339207 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.339194 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" event={"ID":"93642a17-3a9f-4321-ad02-4d9c26fa42c2","Type":"ContainerDied","Data":"cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89"} Nov 03 23:07:18.339836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.339238 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb" event={"ID":"93642a17-3a9f-4321-ad02-4d9c26fa42c2","Type":"ContainerDied","Data":"45985071e41afa41a74f1486f60eec03f48263fa8ae95bd49c3b9f7325be8fbd"} Nov 03 23:07:18.339836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.339262 4903 scope.go:122] "RemoveContainer" containerID="cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89" Nov 03 23:07:18.349928 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.349901 4903 scope.go:122] "RemoveContainer" containerID="cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89" Nov 03 23:07:18.350197 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:07:18.350179 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89\": container with ID starting with cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89 not found: ID does not exist" containerID="cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89" Nov 03 23:07:18.350269 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.350208 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89"} err="failed to get container status \"cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89\": rpc error: code = NotFound desc = could not find container \"cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89\": container with ID starting with cd413dff7bc052fa55ea65054934f6ee705614ac08f148a0b9461a2a2bddeb89 not found: ID does not exist" Nov 03 23:07:18.382282 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:07:18.382253 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:07:18.386887 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.386859 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb"] Nov 03 23:07:18.392878 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:18.392850 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d8a3d1-openshift-default-657d6f85c7-xcxdb"] Nov 03 23:07:19.732522 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:19.732481 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" path="/var/lib/kubelet/pods/93642a17-3a9f-4321-ad02-4d9c26fa42c2/volumes" Nov 03 23:07:20.960180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:20.960138 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" podUID="fddc0ee5-3581-450e-b82a-ca28c12cd15e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:07:23.835413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:23.835371 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml"] Nov 03 23:07:23.866568 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:23.866532 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh"] Nov 03 23:07:24.179329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179280 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh"] Nov 03 23:07:24.179764 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179748 4903 cpu_manager.go:414] "containerMap: removing container" podUID="8620ffe7-06ca-4e7c-8a28-aa8da0726cc9" containerName="revoke-keys" Nov 03 23:07:24.179868 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179766 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8620ffe7-06ca-4e7c-8a28-aa8da0726cc9" containerName="revoke-keys" Nov 03 23:07:24.179868 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179791 4903 cpu_manager.go:414] "containerMap: removing container" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerName="istio-proxy" Nov 03 23:07:24.179868 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179796 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerName="istio-proxy" Nov 03 23:07:24.179868 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179806 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" Nov 03 23:07:24.179868 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179811 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" Nov 03 23:07:24.180146 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179885 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c164f82e-0816-4e68-a013-aea385eb9b83" containerName="main" Nov 03 23:07:24.180146 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179899 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="8620ffe7-06ca-4e7c-8a28-aa8da0726cc9" containerName="revoke-keys" Nov 03 23:07:24.180146 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.179907 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="93642a17-3a9f-4321-ad02-4d9c26fa42c2" containerName="istio-proxy" Nov 03 23:07:24.183756 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.183731 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.199510 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.199483 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:07:24.200691 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.200668 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:07:24.200798 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.200784 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:07:24.200877 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.200869 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:07:24.203330 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.203273 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh"] Nov 03 23:07:24.289161 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.289135 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 23:07:24.338395 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.338364 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksjct\" (UniqueName: \"kubernetes.io/projected/5d987314-2b17-44cc-be49-142bcf987e1c-kube-api-access-ksjct\") pod \"maas-api-revoke-sub-73e709c5-gwfzh\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.338572 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.338413 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e709c5-gwfzh\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.338621 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.338565 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e709c5-gwfzh\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.386816 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.386735 4903 generic.go:356] "Generic (PLEG): container finished" podID="fddc0ee5-3581-450e-b82a-ca28c12cd15e" containerID="5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30" exitCode=0 Nov 03 23:07:24.386816 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.386783 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" Nov 03 23:07:24.387025 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.386822 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" event={"ID":"fddc0ee5-3581-450e-b82a-ca28c12cd15e","Type":"ContainerDied","Data":"5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30"} Nov 03 23:07:24.387025 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.386875 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t" event={"ID":"fddc0ee5-3581-450e-b82a-ca28c12cd15e","Type":"ContainerDied","Data":"d399ce6d9f7b08cbf4feb8e96f98b26340faf2b35fa5e0c6e2f624f56d629348"} Nov 03 23:07:24.387025 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.386899 4903 scope.go:122] "RemoveContainer" containerID="5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30" Nov 03 23:07:24.396885 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.396867 4903 scope.go:122] "RemoveContainer" containerID="5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30" Nov 03 23:07:24.397150 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:07:24.397131 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30\": container with ID starting with 5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30 not found: ID does not exist" containerID="5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30" Nov 03 23:07:24.397202 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.397160 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30"} err="failed to get container status \"5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30\": rpc error: code = NotFound desc = could not find container \"5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30\": container with ID starting with 5f5d34bf38780ff1fceba40699cd1fcdd24a42260007bd44db61804d10227e30 not found: ID does not exist" Nov 03 23:07:24.439850 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.439814 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fddc0ee5-3581-450e-b82a-ca28c12cd15e-kube-api-access-pkhjk\" (UniqueName: \"kubernetes.io/projected/fddc0ee5-3581-450e-b82a-ca28c12cd15e-kube-api-access-pkhjk\") pod \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\" (UID: \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\") " Nov 03 23:07:24.440051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.439949 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fddc0ee5-3581-450e-b82a-ca28c12cd15e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc0ee5-3581-450e-b82a-ca28c12cd15e-plugins-config-volume\") pod \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\" (UID: \"fddc0ee5-3581-450e-b82a-ca28c12cd15e\") " Nov 03 23:07:24.440110 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.440091 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e709c5-gwfzh\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.440188 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.440168 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e709c5-gwfzh\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.440315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.440265 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksjct\" (UniqueName: \"kubernetes.io/projected/5d987314-2b17-44cc-be49-142bcf987e1c-kube-api-access-ksjct\") pod \"maas-api-revoke-sub-73e709c5-gwfzh\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.440465 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.440346 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fddc0ee5-3581-450e-b82a-ca28c12cd15e-plugins-config-volume" pod "fddc0ee5-3581-450e-b82a-ca28c12cd15e" (UID: "fddc0ee5-3581-450e-b82a-ca28c12cd15e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:24.440924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.440902 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73e709c5-gwfzh\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.442223 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.442202 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fddc0ee5-3581-450e-b82a-ca28c12cd15e-kube-api-access-pkhjk" pod "fddc0ee5-3581-450e-b82a-ca28c12cd15e" (UID: "fddc0ee5-3581-450e-b82a-ca28c12cd15e"). InnerVolumeSpecName "kube-api-access-pkhjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:24.442526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.442509 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73e709c5-gwfzh\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.467329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.462984 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksjct\" (UniqueName: \"kubernetes.io/projected/5d987314-2b17-44cc-be49-142bcf987e1c-kube-api-access-ksjct\") pod \"maas-api-revoke-sub-73e709c5-gwfzh\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.515549 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.515508 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:24.542839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.542806 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fddc0ee5-3581-450e-b82a-ca28c12cd15e-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:24.542839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.542840 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkhjk\" (UniqueName: \"kubernetes.io/projected/fddc0ee5-3581-450e-b82a-ca28c12cd15e-kube-api-access-pkhjk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:24.731851 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.731816 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t"] Nov 03 23:07:24.734556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.734532 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d8a3d1-b7986f8f9-s9n6t"] Nov 03 23:07:24.829856 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.829824 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh"] Nov 03 23:07:24.831500 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:07:24.831474 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d987314_2b17_44cc_be49_142bcf987e1c.slice/crio-bc55625b8125c45615ce899e406514c70ed9e6a3c8d678e162d16136a85a9843 WatchSource:0}: Error finding container bc55625b8125c45615ce899e406514c70ed9e6a3c8d678e162d16136a85a9843: Status 404 returned error can't find the container with id bc55625b8125c45615ce899e406514c70ed9e6a3c8d678e162d16136a85a9843 Nov 03 23:07:24.908288 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.908216 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx"] Nov 03 23:07:24.910843 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:24.910819 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-88a4d9-ndxvx"] Nov 03 23:07:25.399204 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:25.399162 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" event={"ID":"5d987314-2b17-44cc-be49-142bcf987e1c","Type":"ContainerStarted","Data":"1582dc0de8f8832c65ab7de18e873996f1ba9d287f611e0cff2098072b56e725"} Nov 03 23:07:25.399442 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:25.399265 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" event={"ID":"5d987314-2b17-44cc-be49-142bcf987e1c","Type":"ContainerStarted","Data":"bc55625b8125c45615ce899e406514c70ed9e6a3c8d678e162d16136a85a9843"} Nov 03 23:07:25.442806 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:25.442679 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" podStartSLOduration=1.442664432 podStartE2EDuration="1.442664432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:07:25.441671687 +0000 UTC m=+3271.845612663" watchObservedRunningTime="2026-11-03 23:07:25.442664432 +0000 UTC m=+3271.846605409" Nov 03 23:07:25.732111 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:25.732034 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a72a922-9d79-4e13-a951-13cf1a99abe1" path="/var/lib/kubelet/pods/4a72a922-9d79-4e13-a951-13cf1a99abe1/volumes" Nov 03 23:07:25.732507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:25.732492 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fddc0ee5-3581-450e-b82a-ca28c12cd15e" path="/var/lib/kubelet/pods/fddc0ee5-3581-450e-b82a-ca28c12cd15e/volumes" Nov 03 23:07:26.411971 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:26.411933 4903 generic.go:356] "Generic (PLEG): container finished" podID="5d987314-2b17-44cc-be49-142bcf987e1c" containerID="1582dc0de8f8832c65ab7de18e873996f1ba9d287f611e0cff2098072b56e725" exitCode=0 Nov 03 23:07:26.412425 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:26.412011 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" event={"ID":"5d987314-2b17-44cc-be49-142bcf987e1c","Type":"ContainerDied","Data":"1582dc0de8f8832c65ab7de18e873996f1ba9d287f611e0cff2098072b56e725"} Nov 03 23:07:27.560101 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.560076 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:27.683164 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.683076 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-openshift-ca\") pod \"5d987314-2b17-44cc-be49-142bcf987e1c\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " Nov 03 23:07:27.683365 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.683217 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-xks-ca\") pod \"5d987314-2b17-44cc-be49-142bcf987e1c\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " Nov 03 23:07:27.683365 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.683279 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d987314-2b17-44cc-be49-142bcf987e1c-kube-api-access-ksjct\" (UniqueName: \"kubernetes.io/projected/5d987314-2b17-44cc-be49-142bcf987e1c-kube-api-access-ksjct\") pod \"5d987314-2b17-44cc-be49-142bcf987e1c\" (UID: \"5d987314-2b17-44cc-be49-142bcf987e1c\") " Nov 03 23:07:27.683522 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.683498 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-openshift-ca" pod "5d987314-2b17-44cc-be49-142bcf987e1c" (UID: "5d987314-2b17-44cc-be49-142bcf987e1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:27.683691 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.683668 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:27.685440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.685417 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-xks-ca" pod "5d987314-2b17-44cc-be49-142bcf987e1c" (UID: "5d987314-2b17-44cc-be49-142bcf987e1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:27.685552 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.685450 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d987314-2b17-44cc-be49-142bcf987e1c-kube-api-access-ksjct" pod "5d987314-2b17-44cc-be49-142bcf987e1c" (UID: "5d987314-2b17-44cc-be49-142bcf987e1c"). InnerVolumeSpecName "kube-api-access-ksjct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:27.785226 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.785190 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksjct\" (UniqueName: \"kubernetes.io/projected/5d987314-2b17-44cc-be49-142bcf987e1c-kube-api-access-ksjct\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:27.785226 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:27.785223 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d987314-2b17-44cc-be49-142bcf987e1c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:28.426753 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:28.426722 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" Nov 03 23:07:28.426941 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:28.426763 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh" event={"ID":"5d987314-2b17-44cc-be49-142bcf987e1c","Type":"ContainerDied","Data":"bc55625b8125c45615ce899e406514c70ed9e6a3c8d678e162d16136a85a9843"} Nov 03 23:07:28.426941 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:28.426812 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc55625b8125c45615ce899e406514c70ed9e6a3c8d678e162d16136a85a9843" Nov 03 23:07:28.836419 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:28.836336 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" podUID="bdf4c6af-f700-4a03-8491-a5e51fd4dab2" containerName="payload-processing" containerID="cri-o://f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250" gracePeriod=25 Nov 03 23:07:28.866973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:28.866942 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" podUID="46d2aab3-c989-42f3-af29-088f1a463ae0" containerName="payload-pre-processing" containerID="cri-o://c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50" gracePeriod=25 Nov 03 23:07:29.130855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.130827 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 23:07:29.158758 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.158735 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 23:07:29.200875 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.200844 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d2aab3-c989-42f3-af29-088f1a463ae0-kube-api-access-28j27\" (UniqueName: \"kubernetes.io/projected/46d2aab3-c989-42f3-af29-088f1a463ae0-kube-api-access-28j27\") pod \"46d2aab3-c989-42f3-af29-088f1a463ae0\" (UID: \"46d2aab3-c989-42f3-af29-088f1a463ae0\") " Nov 03 23:07:29.201038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.200882 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-kube-api-access-p54mr\" (UniqueName: \"kubernetes.io/projected/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-kube-api-access-p54mr\") pod \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\" (UID: \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\") " Nov 03 23:07:29.201038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.200957 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-plugins-config-volume\") pod \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\" (UID: \"bdf4c6af-f700-4a03-8491-a5e51fd4dab2\") " Nov 03 23:07:29.201038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.201009 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d2aab3-c989-42f3-af29-088f1a463ae0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2aab3-c989-42f3-af29-088f1a463ae0-plugins-config-volume\") pod \"46d2aab3-c989-42f3-af29-088f1a463ae0\" (UID: \"46d2aab3-c989-42f3-af29-088f1a463ae0\") " Nov 03 23:07:29.201420 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.201394 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-plugins-config-volume" pod "bdf4c6af-f700-4a03-8491-a5e51fd4dab2" (UID: "bdf4c6af-f700-4a03-8491-a5e51fd4dab2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:29.201508 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.201418 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d2aab3-c989-42f3-af29-088f1a463ae0-plugins-config-volume" pod "46d2aab3-c989-42f3-af29-088f1a463ae0" (UID: "46d2aab3-c989-42f3-af29-088f1a463ae0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:29.203220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.203197 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d2aab3-c989-42f3-af29-088f1a463ae0-kube-api-access-28j27" pod "46d2aab3-c989-42f3-af29-088f1a463ae0" (UID: "46d2aab3-c989-42f3-af29-088f1a463ae0"). InnerVolumeSpecName "kube-api-access-28j27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:29.203353 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.203227 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-kube-api-access-p54mr" pod "bdf4c6af-f700-4a03-8491-a5e51fd4dab2" (UID: "bdf4c6af-f700-4a03-8491-a5e51fd4dab2"). InnerVolumeSpecName "kube-api-access-p54mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:29.301914 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.301878 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28j27\" (UniqueName: \"kubernetes.io/projected/46d2aab3-c989-42f3-af29-088f1a463ae0-kube-api-access-28j27\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:29.301914 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.301907 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p54mr\" (UniqueName: \"kubernetes.io/projected/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-kube-api-access-p54mr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:29.301914 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.301919 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdf4c6af-f700-4a03-8491-a5e51fd4dab2-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:29.302164 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.301929 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2aab3-c989-42f3-af29-088f1a463ae0-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:29.434202 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.434166 4903 generic.go:356] "Generic (PLEG): container finished" podID="46d2aab3-c989-42f3-af29-088f1a463ae0" containerID="c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50" exitCode=0 Nov 03 23:07:29.434414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.434215 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" Nov 03 23:07:29.434414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.434253 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" event={"ID":"46d2aab3-c989-42f3-af29-088f1a463ae0","Type":"ContainerDied","Data":"c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50"} Nov 03 23:07:29.434414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.434330 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh" event={"ID":"46d2aab3-c989-42f3-af29-088f1a463ae0","Type":"ContainerDied","Data":"7ac9fd7dec7508f73c840ede3da910e509b4d2e6756d7715294b508f06e308ff"} Nov 03 23:07:29.434414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.434354 4903 scope.go:122] "RemoveContainer" containerID="c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50" Nov 03 23:07:29.435960 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.435934 4903 generic.go:356] "Generic (PLEG): container finished" podID="bdf4c6af-f700-4a03-8491-a5e51fd4dab2" containerID="f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250" exitCode=0 Nov 03 23:07:29.436069 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.435980 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" Nov 03 23:07:29.436069 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.435997 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" event={"ID":"bdf4c6af-f700-4a03-8491-a5e51fd4dab2","Type":"ContainerDied","Data":"f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250"} Nov 03 23:07:29.436069 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.436048 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml" event={"ID":"bdf4c6af-f700-4a03-8491-a5e51fd4dab2","Type":"ContainerDied","Data":"c48056e9048afbd5f36d12630d161ae5bea3a13aaaacec11bb6756bb6c1ff2df"} Nov 03 23:07:29.446942 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.446924 4903 scope.go:122] "RemoveContainer" containerID="c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50" Nov 03 23:07:29.447270 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:07:29.447247 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50\": container with ID starting with c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50 not found: ID does not exist" containerID="c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50" Nov 03 23:07:29.447349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.447279 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50"} err="failed to get container status \"c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50\": rpc error: code = NotFound desc = could not find container \"c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50\": container with ID starting with c42a5b452f3d7573d069e0350e06d4e4220e44e789da6700effb68e5feaf5d50 not found: ID does not exist" Nov 03 23:07:29.447349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.447313 4903 scope.go:122] "RemoveContainer" containerID="f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250" Nov 03 23:07:29.456189 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.456170 4903 scope.go:122] "RemoveContainer" containerID="f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250" Nov 03 23:07:29.456498 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:07:29.456482 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250\": container with ID starting with f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250 not found: ID does not exist" containerID="f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250" Nov 03 23:07:29.456569 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.456505 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250"} err="failed to get container status \"f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250\": rpc error: code = NotFound desc = could not find container \"f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250\": container with ID starting with f8efef751b0a6ae5272174a173210409ee3245caba8e09dcfe04e07a75f29250 not found: ID does not exist" Nov 03 23:07:29.481071 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.481048 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh"] Nov 03 23:07:29.491118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.491097 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5187e1-7968f4cdbc-klfgh"] Nov 03 23:07:29.525289 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.525252 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml"] Nov 03 23:07:29.528904 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.528878 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5187e1-5d69cc969d-9qmml"] Nov 03 23:07:29.627280 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.627242 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:07:29.732893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.732812 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d2aab3-c989-42f3-af29-088f1a463ae0" path="/var/lib/kubelet/pods/46d2aab3-c989-42f3-af29-088f1a463ae0/volumes" Nov 03 23:07:29.733264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:29.733249 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf4c6af-f700-4a03-8491-a5e51fd4dab2" path="/var/lib/kubelet/pods/bdf4c6af-f700-4a03-8491-a5e51fd4dab2/volumes" Nov 03 23:07:33.194834 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.194796 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" containerID="cri-o://bab7f05d07fa935659f170c50c4c670dbcf78c135cfcd0665f25ef050a9caf32" gracePeriod=45 Nov 03 23:07:33.202575 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.202540 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": dial tcp 10.132.0.100:8000: connect: connection refused" Nov 03 23:07:33.223334 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.223246 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" containerID="cri-o://b930e49abd7fdce7dd4747145e4f49735fe81a03bf0f98d5fc30f37ea5f28044" gracePeriod=45 Nov 03 23:07:33.262325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.262281 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6"] Nov 03 23:07:33.262568 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.262544 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" containerID="cri-o://8466bb7eb798cab114c95642fe96942b0743bd92e977d1507587bb3037496e30" gracePeriod=30 Nov 03 23:07:33.470476 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.470440 4903 generic.go:356] "Generic (PLEG): container finished" podID="eac61526-e700-48e0-9293-f561cd920d53" containerID="bab7f05d07fa935659f170c50c4c670dbcf78c135cfcd0665f25ef050a9caf32" exitCode=0 Nov 03 23:07:33.470613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.470513 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" event={"ID":"eac61526-e700-48e0-9293-f561cd920d53","Type":"ContainerDied","Data":"bab7f05d07fa935659f170c50c4c670dbcf78c135cfcd0665f25ef050a9caf32"} Nov 03 23:07:33.472341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.472313 4903 generic.go:356] "Generic (PLEG): container finished" podID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerID="b930e49abd7fdce7dd4747145e4f49735fe81a03bf0f98d5fc30f37ea5f28044" exitCode=0 Nov 03 23:07:33.472524 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.472505 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" event={"ID":"2a063f54-ad04-4e48-8b3c-246b782ce808","Type":"ContainerDied","Data":"b930e49abd7fdce7dd4747145e4f49735fe81a03bf0f98d5fc30f37ea5f28044"} Nov 03 23:07:33.491279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.491261 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 23:07:33.517482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.517462 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 23:07:33.560105 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560071 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-dshm\") pod \"eac61526-e700-48e0-9293-f561cd920d53\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " Nov 03 23:07:33.560279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560111 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eac61526-e700-48e0-9293-f561cd920d53-kube-api-access-hztvp\" (UniqueName: \"kubernetes.io/projected/eac61526-e700-48e0-9293-f561cd920d53-kube-api-access-hztvp\") pod \"eac61526-e700-48e0-9293-f561cd920d53\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " Nov 03 23:07:33.560279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560144 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a063f54-ad04-4e48-8b3c-246b782ce808-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a063f54-ad04-4e48-8b3c-246b782ce808-tls-certs\") pod \"2a063f54-ad04-4e48-8b3c-246b782ce808\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " Nov 03 23:07:33.560279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560220 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-tmp-dir\") pod \"eac61526-e700-48e0-9293-f561cd920d53\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " Nov 03 23:07:33.560279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560267 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-tmp-dir\") pod \"2a063f54-ad04-4e48-8b3c-246b782ce808\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " Nov 03 23:07:33.560556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560320 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-dshm\") pod \"2a063f54-ad04-4e48-8b3c-246b782ce808\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " Nov 03 23:07:33.560556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560354 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-home\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-home\") pod \"2a063f54-ad04-4e48-8b3c-246b782ce808\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " Nov 03 23:07:33.560556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560419 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-home\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-home\") pod \"eac61526-e700-48e0-9293-f561cd920d53\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " Nov 03 23:07:33.560556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560464 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a063f54-ad04-4e48-8b3c-246b782ce808-kube-api-access-mvn8d\" (UniqueName: \"kubernetes.io/projected/2a063f54-ad04-4e48-8b3c-246b782ce808-kube-api-access-mvn8d\") pod \"2a063f54-ad04-4e48-8b3c-246b782ce808\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " Nov 03 23:07:33.560556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560525 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eac61526-e700-48e0-9293-f561cd920d53-tls-certs\" (UniqueName: \"kubernetes.io/secret/eac61526-e700-48e0-9293-f561cd920d53-tls-certs\") pod \"eac61526-e700-48e0-9293-f561cd920d53\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " Nov 03 23:07:33.560836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560588 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-model-cache\") pod \"eac61526-e700-48e0-9293-f561cd920d53\" (UID: \"eac61526-e700-48e0-9293-f561cd920d53\") " Nov 03 23:07:33.560836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.560614 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-model-cache\") pod \"2a063f54-ad04-4e48-8b3c-246b782ce808\" (UID: \"2a063f54-ad04-4e48-8b3c-246b782ce808\") " Nov 03 23:07:33.561819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.561221 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-tmp-dir" pod "2a063f54-ad04-4e48-8b3c-246b782ce808" (UID: "2a063f54-ad04-4e48-8b3c-246b782ce808"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:33.561819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.561235 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-model-cache" pod "2a063f54-ad04-4e48-8b3c-246b782ce808" (UID: "2a063f54-ad04-4e48-8b3c-246b782ce808"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:33.561819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.561481 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-tmp-dir" pod "eac61526-e700-48e0-9293-f561cd920d53" (UID: "eac61526-e700-48e0-9293-f561cd920d53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:33.561819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.561627 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-home" pod "eac61526-e700-48e0-9293-f561cd920d53" (UID: "eac61526-e700-48e0-9293-f561cd920d53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:33.561819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.561699 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-model-cache" pod "eac61526-e700-48e0-9293-f561cd920d53" (UID: "eac61526-e700-48e0-9293-f561cd920d53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:33.561819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.561708 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-home" pod "2a063f54-ad04-4e48-8b3c-246b782ce808" (UID: "2a063f54-ad04-4e48-8b3c-246b782ce808"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:33.563068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.563041 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a063f54-ad04-4e48-8b3c-246b782ce808-tls-certs" pod "2a063f54-ad04-4e48-8b3c-246b782ce808" (UID: "2a063f54-ad04-4e48-8b3c-246b782ce808"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:33.563068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.563059 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-dshm" pod "eac61526-e700-48e0-9293-f561cd920d53" (UID: "eac61526-e700-48e0-9293-f561cd920d53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:33.563287 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.563263 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac61526-e700-48e0-9293-f561cd920d53-kube-api-access-hztvp" pod "eac61526-e700-48e0-9293-f561cd920d53" (UID: "eac61526-e700-48e0-9293-f561cd920d53"). InnerVolumeSpecName "kube-api-access-hztvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:33.563995 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.563969 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-dshm" pod "2a063f54-ad04-4e48-8b3c-246b782ce808" (UID: "2a063f54-ad04-4e48-8b3c-246b782ce808"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:33.564469 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.564449 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac61526-e700-48e0-9293-f561cd920d53-tls-certs" pod "eac61526-e700-48e0-9293-f561cd920d53" (UID: "eac61526-e700-48e0-9293-f561cd920d53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:33.565964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.565941 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a063f54-ad04-4e48-8b3c-246b782ce808-kube-api-access-mvn8d" pod "2a063f54-ad04-4e48-8b3c-246b782ce808" (UID: "2a063f54-ad04-4e48-8b3c-246b782ce808"). InnerVolumeSpecName "kube-api-access-mvn8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:33.662123 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662091 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eac61526-e700-48e0-9293-f561cd920d53-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662123 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662121 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662123 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662130 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662139 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662148 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hztvp\" (UniqueName: \"kubernetes.io/projected/eac61526-e700-48e0-9293-f561cd920d53-kube-api-access-hztvp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662158 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a063f54-ad04-4e48-8b3c-246b782ce808-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662166 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662173 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662180 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662188 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a063f54-ad04-4e48-8b3c-246b782ce808-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662195 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eac61526-e700-48e0-9293-f561cd920d53-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:33.662385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:33.662202 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvn8d\" (UniqueName: \"kubernetes.io/projected/2a063f54-ad04-4e48-8b3c-246b782ce808-kube-api-access-mvn8d\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:34.454777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.454734 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 23:07:34.480850 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.480766 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" Nov 03 23:07:34.480850 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.480790 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk" event={"ID":"eac61526-e700-48e0-9293-f561cd920d53","Type":"ContainerDied","Data":"b20d5a9d825f1d4f957443ac955776ff92f4d638cf77416540190450d52ad9ba"} Nov 03 23:07:34.480850 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.480845 4903 scope.go:122] "RemoveContainer" containerID="bab7f05d07fa935659f170c50c4c670dbcf78c135cfcd0665f25ef050a9caf32" Nov 03 23:07:34.482279 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.482256 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" event={"ID":"2a063f54-ad04-4e48-8b3c-246b782ce808","Type":"ContainerDied","Data":"52b374986bd65f9e85ca088bb8d93e6be9f3fedae293a874491a627927450d93"} Nov 03 23:07:34.482389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.482270 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5" Nov 03 23:07:34.490607 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.490590 4903 scope.go:122] "RemoveContainer" containerID="b930e49abd7fdce7dd4747145e4f49735fe81a03bf0f98d5fc30f37ea5f28044" Nov 03 23:07:34.526942 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.526916 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk"] Nov 03 23:07:34.536974 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.536950 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/distinct-2-w3-5187e1-kserve-bf5df55dd-br7bk"] Nov 03 23:07:34.559722 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.559700 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5"] Nov 03 23:07:34.563721 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:34.563702 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5187e1/simulator-w3-5187e1-kserve-657d5cc745-57hv5"] Nov 03 23:07:35.732956 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:35.732921 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" path="/var/lib/kubelet/pods/2a063f54-ad04-4e48-8b3c-246b782ce808/volumes" Nov 03 23:07:35.733476 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:35.733375 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eac61526-e700-48e0-9293-f561cd920d53" path="/var/lib/kubelet/pods/eac61526-e700-48e0-9293-f561cd920d53/volumes" Nov 03 23:07:36.362751 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:36.362714 4903 patch_prober.go:29] interesting pod/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:07:36.362937 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:36.362781 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:07:36.937937 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:36.937903 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm"] Nov 03 23:07:36.947834 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:36.947804 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5187e1-mf7zm"] Nov 03 23:07:37.731369 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:37.731339 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8620ffe7-06ca-4e7c-8a28-aa8da0726cc9" path="/var/lib/kubelet/pods/8620ffe7-06ca-4e7c-8a28-aa8da0726cc9/volumes" Nov 03 23:07:38.513975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.513940 4903 generic.go:356] "Generic (PLEG): container finished" podID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerID="8466bb7eb798cab114c95642fe96942b0743bd92e977d1507587bb3037496e30" exitCode=0 Nov 03 23:07:38.514385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.514009 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" event={"ID":"db522d78-bd16-46c0-8dbd-f6670cb42c65","Type":"ContainerDied","Data":"8466bb7eb798cab114c95642fe96942b0743bd92e977d1507587bb3037496e30"} Nov 03 23:07:38.533233 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.533163 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 23:07:38.610229 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610194 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-credential-socket\") pod \"db522d78-bd16-46c0-8dbd-f6670cb42c65\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " Nov 03 23:07:38.610229 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610234 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-podinfo\") pod \"db522d78-bd16-46c0-8dbd-f6670cb42c65\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " Nov 03 23:07:38.610501 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610265 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db522d78-bd16-46c0-8dbd-f6670cb42c65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db522d78-bd16-46c0-8dbd-f6670cb42c65-istiod-ca-cert\") pod \"db522d78-bd16-46c0-8dbd-f6670cb42c65\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " Nov 03 23:07:38.610501 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610342 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-certs\") pod \"db522d78-bd16-46c0-8dbd-f6670cb42c65\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " Nov 03 23:07:38.610501 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610419 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-socket\") pod \"db522d78-bd16-46c0-8dbd-f6670cb42c65\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " Nov 03 23:07:38.610501 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610484 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-data\") pod \"db522d78-bd16-46c0-8dbd-f6670cb42c65\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " Nov 03 23:07:38.610739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610543 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-credential-socket" pod "db522d78-bd16-46c0-8dbd-f6670cb42c65" (UID: "db522d78-bd16-46c0-8dbd-f6670cb42c65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:38.610739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610564 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-kube-api-access-4pl48\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-kube-api-access-4pl48\") pod \"db522d78-bd16-46c0-8dbd-f6670cb42c65\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " Nov 03 23:07:38.610739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610596 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-token\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-token\") pod \"db522d78-bd16-46c0-8dbd-f6670cb42c65\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " Nov 03 23:07:38.610739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610633 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-envoy\") pod \"db522d78-bd16-46c0-8dbd-f6670cb42c65\" (UID: \"db522d78-bd16-46c0-8dbd-f6670cb42c65\") " Nov 03 23:07:38.610739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610643 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-certs" pod "db522d78-bd16-46c0-8dbd-f6670cb42c65" (UID: "db522d78-bd16-46c0-8dbd-f6670cb42c65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:38.610739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610666 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-socket" pod "db522d78-bd16-46c0-8dbd-f6670cb42c65" (UID: "db522d78-bd16-46c0-8dbd-f6670cb42c65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:38.611089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610758 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-data" pod "db522d78-bd16-46c0-8dbd-f6670cb42c65" (UID: "db522d78-bd16-46c0-8dbd-f6670cb42c65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:38.611089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610807 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db522d78-bd16-46c0-8dbd-f6670cb42c65-istiod-ca-cert" pod "db522d78-bd16-46c0-8dbd-f6670cb42c65" (UID: "db522d78-bd16-46c0-8dbd-f6670cb42c65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:38.611089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.610985 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:38.611089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.611007 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:38.611089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.611021 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:38.611089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.611035 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:38.611089 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.611050 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db522d78-bd16-46c0-8dbd-f6670cb42c65-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:38.612811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.612783 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-token" pod "db522d78-bd16-46c0-8dbd-f6670cb42c65" (UID: "db522d78-bd16-46c0-8dbd-f6670cb42c65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:38.612926 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.612791 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-podinfo" pod "db522d78-bd16-46c0-8dbd-f6670cb42c65" (UID: "db522d78-bd16-46c0-8dbd-f6670cb42c65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:38.612926 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.612854 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-envoy" pod "db522d78-bd16-46c0-8dbd-f6670cb42c65" (UID: "db522d78-bd16-46c0-8dbd-f6670cb42c65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:38.613913 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.613888 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-kube-api-access-4pl48" pod "db522d78-bd16-46c0-8dbd-f6670cb42c65" (UID: "db522d78-bd16-46c0-8dbd-f6670cb42c65"). InnerVolumeSpecName "kube-api-access-4pl48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:38.712336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.712287 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pl48\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-kube-api-access-4pl48\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:38.712336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.712335 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:38.712336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.712347 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:38.712589 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:38.712355 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db522d78-bd16-46c0-8dbd-f6670cb42c65-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:39.523107 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:39.523071 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" event={"ID":"db522d78-bd16-46c0-8dbd-f6670cb42c65","Type":"ContainerDied","Data":"dc6078017b3322711da864e2aa00be560a2df8afad5fe93802bf0ff320f6e7af"} Nov 03 23:07:39.523605 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:39.523133 4903 scope.go:122] "RemoveContainer" containerID="8466bb7eb798cab114c95642fe96942b0743bd92e977d1507587bb3037496e30" Nov 03 23:07:39.523605 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:39.523085 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6" Nov 03 23:07:39.579639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:39.579604 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6"] Nov 03 23:07:39.589463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:39.589436 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-88a4d9-openshift-default-857757469c-8lzn6"] Nov 03 23:07:39.731331 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:39.731274 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" path="/var/lib/kubelet/pods/db522d78-bd16-46c0-8dbd-f6670cb42c65/volumes" Nov 03 23:07:42.027949 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:42.027914 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2"] Nov 03 23:07:42.057081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:42.038620 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9456c75-wvgf2"] Nov 03 23:07:42.487442 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:42.487407 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:07:43.732049 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:43.732016 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="791668bf-7e0e-4569-8e40-f0a87c2d6b55" path="/var/lib/kubelet/pods/791668bf-7e0e-4569-8e40-f0a87c2d6b55/volumes" Nov 03 23:07:44.736535 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:44.736500 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd"] Nov 03 23:07:44.737036 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:44.736752 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" containerID="cri-o://5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd" gracePeriod=30 Nov 03 23:07:48.385994 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:07:48.385956 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:07:49.492989 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.492957 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9589f759-8srgp"] Nov 03 23:07:49.493463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.493181 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6d9589f759-8srgp" podUID="f40c3a4f-13bb-4d9a-a3b6-9ae260278d88" containerName="manager" containerID="cri-o://029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79" gracePeriod=30 Nov 03 23:07:49.757930 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.757907 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 23:07:49.933814 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.933785 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-webhook-cert\") pod \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " Nov 03 23:07:49.933982 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.933845 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-kube-api-access-8t2km\" (UniqueName: \"kubernetes.io/projected/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-kube-api-access-8t2km\") pod \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " Nov 03 23:07:49.933982 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.933867 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-metrics-certs\") pod \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\" (UID: \"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88\") " Nov 03 23:07:49.936235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.936179 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-webhook-cert" pod "f40c3a4f-13bb-4d9a-a3b6-9ae260278d88" (UID: "f40c3a4f-13bb-4d9a-a3b6-9ae260278d88"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:49.936235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.936192 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-kube-api-access-8t2km" pod "f40c3a4f-13bb-4d9a-a3b6-9ae260278d88" (UID: "f40c3a4f-13bb-4d9a-a3b6-9ae260278d88"). InnerVolumeSpecName "kube-api-access-8t2km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:49.936474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.936450 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-metrics-certs" pod "f40c3a4f-13bb-4d9a-a3b6-9ae260278d88" (UID: "f40c3a4f-13bb-4d9a-a3b6-9ae260278d88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:07:49.980948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:49.980923 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 23:07:50.034880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.034809 4903 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-webhook-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.034880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.034839 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t2km\" (UniqueName: \"kubernetes.io/projected/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-kube-api-access-8t2km\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.034880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.034849 4903 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88-metrics-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.135526 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135492 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-socket\") pod \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " Nov 03 23:07:50.135702 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135539 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-envoy\") pod \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " Nov 03 23:07:50.135702 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135560 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-token\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-token\") pod \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " Nov 03 23:07:50.135702 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135592 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-certs\") pod \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " Nov 03 23:07:50.135702 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135627 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istiod-ca-cert\") pod \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " Nov 03 23:07:50.135702 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135649 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-kube-api-access-z58ct\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-kube-api-access-z58ct\") pod \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " Nov 03 23:07:50.135981 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135771 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-credential-socket\") pod \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " Nov 03 23:07:50.135981 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135837 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-data\") pod \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " Nov 03 23:07:50.135981 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135842 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-socket" pod "aa58306f-409e-4dfb-8723-2f1ae4e66d18" (UID: "aa58306f-409e-4dfb-8723-2f1ae4e66d18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:50.135981 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135893 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-podinfo\") pod \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\" (UID: \"aa58306f-409e-4dfb-8723-2f1ae4e66d18\") " Nov 03 23:07:50.135981 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.135915 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-certs" pod "aa58306f-409e-4dfb-8723-2f1ae4e66d18" (UID: "aa58306f-409e-4dfb-8723-2f1ae4e66d18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:50.136241 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.136096 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-credential-socket" pod "aa58306f-409e-4dfb-8723-2f1ae4e66d18" (UID: "aa58306f-409e-4dfb-8723-2f1ae4e66d18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:50.136241 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.136109 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-data" pod "aa58306f-409e-4dfb-8723-2f1ae4e66d18" (UID: "aa58306f-409e-4dfb-8723-2f1ae4e66d18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:50.136241 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.136128 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istiod-ca-cert" pod "aa58306f-409e-4dfb-8723-2f1ae4e66d18" (UID: "aa58306f-409e-4dfb-8723-2f1ae4e66d18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:07:50.136459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.136288 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.136459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.136333 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.136459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.136348 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.136459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.136362 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.136459 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.136375 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.137777 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.137756 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-envoy" pod "aa58306f-409e-4dfb-8723-2f1ae4e66d18" (UID: "aa58306f-409e-4dfb-8723-2f1ae4e66d18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:07:50.137862 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.137840 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-token" pod "aa58306f-409e-4dfb-8723-2f1ae4e66d18" (UID: "aa58306f-409e-4dfb-8723-2f1ae4e66d18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:50.138384 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.138357 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-kube-api-access-z58ct" pod "aa58306f-409e-4dfb-8723-2f1ae4e66d18" (UID: "aa58306f-409e-4dfb-8723-2f1ae4e66d18"). InnerVolumeSpecName "kube-api-access-z58ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:07:50.138499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.138424 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-podinfo" pod "aa58306f-409e-4dfb-8723-2f1ae4e66d18" (UID: "aa58306f-409e-4dfb-8723-2f1ae4e66d18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:07:50.236891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.236853 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z58ct\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-kube-api-access-z58ct\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.236891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.236886 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.236891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.236899 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.237179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.236911 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa58306f-409e-4dfb-8723-2f1ae4e66d18-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:07:50.595663 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.595630 4903 generic.go:356] "Generic (PLEG): container finished" podID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerID="5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd" exitCode=0 Nov 03 23:07:50.596109 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.595680 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" Nov 03 23:07:50.596109 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.595719 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" event={"ID":"aa58306f-409e-4dfb-8723-2f1ae4e66d18","Type":"ContainerDied","Data":"5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd"} Nov 03 23:07:50.596109 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.595774 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd" event={"ID":"aa58306f-409e-4dfb-8723-2f1ae4e66d18","Type":"ContainerDied","Data":"fa9ed72ba38451c2376ea54d68370914d72335307fa13e7480dfbde2f6537b6d"} Nov 03 23:07:50.596109 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.595798 4903 scope.go:122] "RemoveContainer" containerID="5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd" Nov 03 23:07:50.597140 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.597122 4903 generic.go:356] "Generic (PLEG): container finished" podID="f40c3a4f-13bb-4d9a-a3b6-9ae260278d88" containerID="029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79" exitCode=0 Nov 03 23:07:50.597224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.597169 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d9589f759-8srgp" Nov 03 23:07:50.597224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.597177 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-8srgp" event={"ID":"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88","Type":"ContainerDied","Data":"029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79"} Nov 03 23:07:50.597361 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.597240 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d9589f759-8srgp" event={"ID":"f40c3a4f-13bb-4d9a-a3b6-9ae260278d88","Type":"ContainerDied","Data":"ee49703ba68be15a51603032e41cf942f5e93497ebff8bd85b521b36c25c1d1c"} Nov 03 23:07:50.607096 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.607071 4903 scope.go:122] "RemoveContainer" containerID="5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd" Nov 03 23:07:50.607399 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:07:50.607373 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd\": container with ID starting with 5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd not found: ID does not exist" containerID="5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd" Nov 03 23:07:50.607482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.607405 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd"} err="failed to get container status \"5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd\": rpc error: code = NotFound desc = could not find container \"5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd\": container with ID starting with 5f8e16b5e749fedd49f30f1b0552cb58836836ca5fe292801fb2606b916683bd not found: ID does not exist" Nov 03 23:07:50.607482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.607424 4903 scope.go:122] "RemoveContainer" containerID="029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79" Nov 03 23:07:50.618566 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.618550 4903 scope.go:122] "RemoveContainer" containerID="029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79" Nov 03 23:07:50.618807 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:07:50.618790 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79\": container with ID starting with 029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79 not found: ID does not exist" containerID="029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79" Nov 03 23:07:50.618845 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.618815 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79"} err="failed to get container status \"029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79\": rpc error: code = NotFound desc = could not find container \"029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79\": container with ID starting with 029b35d0d5b1625bfd74eb3e9087fa3460cb0612825449a690921743edc52f79 not found: ID does not exist" Nov 03 23:07:50.645419 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.645395 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd"] Nov 03 23:07:50.655625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.655599 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5187e1-openshift-default-946849bdf-knhhd"] Nov 03 23:07:50.685324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.685280 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d9589f759-8srgp"] Nov 03 23:07:50.689200 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:50.689178 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d9589f759-8srgp"] Nov 03 23:07:51.731721 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:51.731687 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" path="/var/lib/kubelet/pods/aa58306f-409e-4dfb-8723-2f1ae4e66d18/volumes" Nov 03 23:07:51.732254 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:51.732070 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40c3a4f-13bb-4d9a-a3b6-9ae260278d88" path="/var/lib/kubelet/pods/f40c3a4f-13bb-4d9a-a3b6-9ae260278d88/volumes" Nov 03 23:07:54.428697 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:54.428667 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:07:54.434682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:54.434656 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:07:54.436828 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:54.436799 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:07:54.442807 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:07:54.442777 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:08:00.549806 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:08:00.549778 4903 scope.go:122] "RemoveContainer" containerID="d99fd53d1b038b7eae0440728e0ceba5c54f94aa75ac73f9d70b7656f07494c6" Nov 03 23:08:00.559325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:08:00.559282 4903 scope.go:122] "RemoveContainer" containerID="4a465f803943fd48401addecc14fbfadaaeedfa2e1de81f54c15a4cf9074e2dc" Nov 03 23:08:00.568224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:08:00.568206 4903 scope.go:122] "RemoveContainer" containerID="a288db4429a55837846d2925455f32efe028670c0f739f3846ef05ecdfab213c" Nov 03 23:08:00.576555 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:08:00.576535 4903 scope.go:122] "RemoveContainer" containerID="4945c03ff3c70623472fdf81a58be294307ed2f00de4464b0c31ae6fe3401c69" Nov 03 23:08:18.383386 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:08:18.383337 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:08:48.385569 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:08:48.385526 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:09:00.663148 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:00.663113 4903 scope.go:122] "RemoveContainer" containerID="5d48370d7ff06aacd96a24f5b0f9f27bdfee87b8f90d26c15a135370b1d245cc" Nov 03 23:09:00.672622 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:00.672599 4903 scope.go:122] "RemoveContainer" containerID="738d2823840ccc9014d2513d01c0b6692b2a74470568d4815c45b21a015e8f17" Nov 03 23:09:00.681197 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:00.681180 4903 scope.go:122] "RemoveContainer" containerID="57d8df15e73b241f5e3bb95b9f05adb4dfb871e17271cd4045c2bfc3229a4dbb" Nov 03 23:09:00.689486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:00.689465 4903 scope.go:122] "RemoveContainer" containerID="3bf0ed812a1c5f3430064cbadb285a86a8d37a820eb0978ede838d89e64ec9f4" Nov 03 23:09:05.409987 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:05.409951 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:09:18.383248 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:09:18.383167 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:09:22.404978 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:22.404943 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:09:27.023797 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:27.023762 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:09:38.205763 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.205732 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql"] Nov 03 23:09:38.206274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206247 4903 cpu_manager.go:414] "containerMap: removing container" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" Nov 03 23:09:38.206274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206270 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206284 4903 cpu_manager.go:414] "containerMap: removing container" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206289 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206321 4903 cpu_manager.go:414] "containerMap: removing container" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206327 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206336 4903 cpu_manager.go:414] "containerMap: removing container" podUID="bdf4c6af-f700-4a03-8491-a5e51fd4dab2" containerName="payload-processing" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206342 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf4c6af-f700-4a03-8491-a5e51fd4dab2" containerName="payload-processing" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206356 4903 cpu_manager.go:414] "containerMap: removing container" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206360 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206377 4903 cpu_manager.go:414] "containerMap: removing container" podUID="fddc0ee5-3581-450e-b82a-ca28c12cd15e" containerName="payload-pre-processing" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206382 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fddc0ee5-3581-450e-b82a-ca28c12cd15e" containerName="payload-pre-processing" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206391 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5d987314-2b17-44cc-be49-142bcf987e1c" containerName="revoke-keys" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206395 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d987314-2b17-44cc-be49-142bcf987e1c" containerName="revoke-keys" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206403 4903 cpu_manager.go:414] "containerMap: removing container" podUID="f40c3a4f-13bb-4d9a-a3b6-9ae260278d88" containerName="manager" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206408 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40c3a4f-13bb-4d9a-a3b6-9ae260278d88" containerName="manager" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206413 4903 cpu_manager.go:414] "containerMap: removing container" podUID="46d2aab3-c989-42f3-af29-088f1a463ae0" containerName="payload-pre-processing" Nov 03 23:09:38.206413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206417 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d2aab3-c989-42f3-af29-088f1a463ae0" containerName="payload-pre-processing" Nov 03 23:09:38.206973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206505 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d987314-2b17-44cc-be49-142bcf987e1c" containerName="revoke-keys" Nov 03 23:09:38.206973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206516 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40c3a4f-13bb-4d9a-a3b6-9ae260278d88" containerName="manager" Nov 03 23:09:38.206973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206521 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d2aab3-c989-42f3-af29-088f1a463ae0" containerName="payload-pre-processing" Nov 03 23:09:38.206973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206530 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="db522d78-bd16-46c0-8dbd-f6670cb42c65" containerName="istio-proxy" Nov 03 23:09:38.206973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206540 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a063f54-ad04-4e48-8b3c-246b782ce808" containerName="main" Nov 03 23:09:38.206973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206549 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="eac61526-e700-48e0-9293-f561cd920d53" containerName="main" Nov 03 23:09:38.206973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206556 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf4c6af-f700-4a03-8491-a5e51fd4dab2" containerName="payload-processing" Nov 03 23:09:38.206973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206563 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa58306f-409e-4dfb-8723-2f1ae4e66d18" containerName="istio-proxy" Nov 03 23:09:38.206973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.206570 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="fddc0ee5-3581-450e-b82a-ca28c12cd15e" containerName="payload-pre-processing" Nov 03 23:09:38.209760 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.209726 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.224576 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.224554 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08263e96-b3ee-4661-9110-4a06dcae57e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.224700 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.224581 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.224700 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.224602 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.224826 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.224678 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.224826 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.224768 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sst5q\" (UniqueName: \"kubernetes.io/projected/08263e96-b3ee-4661-9110-4a06dcae57e3-kube-api-access-sst5q\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.224826 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.224791 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.224972 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.224828 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.224972 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.224895 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.224972 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.224921 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.238757 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.238723 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql"] Nov 03 23:09:38.241100 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.241067 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll"] Nov 03 23:09:38.245455 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.245435 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.263877 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.263816 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll"] Nov 03 23:09:38.326373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326336 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.326373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326372 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.326648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326401 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sst5q\" (UniqueName: \"kubernetes.io/projected/08263e96-b3ee-4661-9110-4a06dcae57e3-kube-api-access-sst5q\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.326648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326512 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.326648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326541 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.326648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326579 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.326648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326599 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.326648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326631 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.326952 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326670 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7gd\" (UniqueName: \"kubernetes.io/projected/8e3a38ee-6f79-46eb-a285-8052c4b0228f-kube-api-access-wx7gd\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.326952 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326701 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.326952 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326805 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.326952 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326877 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.326952 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326938 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.327224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326980 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08263e96-b3ee-4661-9110-4a06dcae57e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.327224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.327009 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.327224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.327025 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.327224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.326984 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.327224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.327031 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.327224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.327080 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.327224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.327113 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.327682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.327280 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.327682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.327404 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.327682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.327550 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08263e96-b3ee-4661-9110-4a06dcae57e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.329030 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.329004 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.329497 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.329477 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.347717 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.347687 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08263e96-b3ee-4661-9110-4a06dcae57e3-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.347825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.347807 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sst5q\" (UniqueName: \"kubernetes.io/projected/08263e96-b3ee-4661-9110-4a06dcae57e3-kube-api-access-sst5q\") pod \"maas-default-gateway-openshift-default-7df854bf67-fkpql\" (UID: \"08263e96-b3ee-4661-9110-4a06dcae57e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.428182 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428143 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7gd\" (UniqueName: \"kubernetes.io/projected/8e3a38ee-6f79-46eb-a285-8052c4b0228f-kube-api-access-wx7gd\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428205 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428230 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428248 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428275 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428337 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428362 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428405 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428455 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428794 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428832 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428944 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428894 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.428944 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.428919 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.429231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.429213 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.430789 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.430762 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.430915 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.430894 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.449860 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.449838 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7gd\" (UniqueName: \"kubernetes.io/projected/8e3a38ee-6f79-46eb-a285-8052c4b0228f-kube-api-access-wx7gd\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.451112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.451093 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3a38ee-6f79-46eb-a285-8052c4b0228f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-f7pll\" (UID: \"8e3a38ee-6f79-46eb-a285-8052c4b0228f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.523077 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.522992 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:38.564128 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.564084 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:38.845318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.845279 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql"] Nov 03 23:09:38.846796 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:09:38.846769 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08263e96_b3ee_4661_9110_4a06dcae57e3.slice/crio-b49a774737cc6a181ebbe25c9bca920dc9ca82eb9287a8f9f75295717b757a34 WatchSource:0}: Error finding container b49a774737cc6a181ebbe25c9bca920dc9ca82eb9287a8f9f75295717b757a34: Status 404 returned error can't find the container with id b49a774737cc6a181ebbe25c9bca920dc9ca82eb9287a8f9f75295717b757a34 Nov 03 23:09:38.848812 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.848782 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:09:38.848889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.848845 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:09:38.848889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.848873 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:09:38.886056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.886037 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll"] Nov 03 23:09:38.887577 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:09:38.887552 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3a38ee_6f79_46eb_a285_8052c4b0228f.slice/crio-dc9469f47cc96dd0a82c131ea6b8c92e77393460f4b138ec7b89957ac3251138 WatchSource:0}: Error finding container dc9469f47cc96dd0a82c131ea6b8c92e77393460f4b138ec7b89957ac3251138: Status 404 returned error can't find the container with id dc9469f47cc96dd0a82c131ea6b8c92e77393460f4b138ec7b89957ac3251138 Nov 03 23:09:38.889819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.889792 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:09:38.889891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.889850 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:09:38.889891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:38.889879 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:09:39.288317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.288141 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" event={"ID":"8e3a38ee-6f79-46eb-a285-8052c4b0228f","Type":"ContainerStarted","Data":"27807e6b7b131d3e12f59b26885a9256cf39acc04fd7c01afff8323c7f00768e"} Nov 03 23:09:39.288317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.288210 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" event={"ID":"8e3a38ee-6f79-46eb-a285-8052c4b0228f","Type":"ContainerStarted","Data":"dc9469f47cc96dd0a82c131ea6b8c92e77393460f4b138ec7b89957ac3251138"} Nov 03 23:09:39.291656 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.291628 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" event={"ID":"08263e96-b3ee-4661-9110-4a06dcae57e3","Type":"ContainerStarted","Data":"0bbd696831a1ba935827160d6792995cefd8bd5bdf50c7677abe77bb71358c94"} Nov 03 23:09:39.291818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.291679 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" event={"ID":"08263e96-b3ee-4661-9110-4a06dcae57e3","Type":"ContainerStarted","Data":"b49a774737cc6a181ebbe25c9bca920dc9ca82eb9287a8f9f75295717b757a34"} Nov 03 23:09:39.333090 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.331723 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podStartSLOduration=1.3317034620000001 podStartE2EDuration="1.331703462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:39.331164381 +0000 UTC m=+3405.735105358" watchObservedRunningTime="2026-11-03 23:09:39.331703462 +0000 UTC m=+3405.735644438" Nov 03 23:09:39.396345 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.385950 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podStartSLOduration=1.385929674 podStartE2EDuration="1.385929674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:09:39.384042533 +0000 UTC m=+3405.787983512" watchObservedRunningTime="2026-11-03 23:09:39.385929674 +0000 UTC m=+3405.789870804" Nov 03 23:09:39.523799 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.523768 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:39.525409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.525386 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" start-of-body= Nov 03 23:09:39.525494 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.525431 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" Nov 03 23:09:39.567377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.567006 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:39.577525 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.577487 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 23:09:39.577857 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:39.577810 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 23:09:40.524449 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:40.524407 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" start-of-body= Nov 03 23:09:40.524880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:40.524473 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" Nov 03 23:09:40.567316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:40.566939 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 23:09:40.567316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:40.567013 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 23:09:41.524148 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:41.524108 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" start-of-body= Nov 03 23:09:41.524417 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:41.524165 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" Nov 03 23:09:41.571316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:41.570976 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 23:09:41.571316 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:41.571044 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 23:09:42.524161 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:42.524128 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" start-of-body= Nov 03 23:09:42.524357 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:42.524183 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" Nov 03 23:09:42.567319 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:42.566692 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 23:09:42.567319 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:42.566753 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 23:09:43.523872 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:43.523834 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" start-of-body= Nov 03 23:09:43.524315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:43.523893 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" Nov 03 23:09:43.569318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:43.568786 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 23:09:43.569318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:43.568863 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 23:09:44.524371 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:44.524333 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" start-of-body= Nov 03 23:09:44.524790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:44.524390 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" Nov 03 23:09:44.566318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:44.565984 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 23:09:44.566318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:44.566058 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 23:09:44.604205 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:44.604169 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:09:45.523613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:45.523575 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" start-of-body= Nov 03 23:09:45.523795 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:45.523643 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": dial tcp 10.132.1.139:15021: connect: connection refused" Nov 03 23:09:45.565549 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:45.565517 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 23:09:45.565920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:45.565581 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 23:09:47.523857 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:47.523812 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:09:47.524274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:47.523888 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:09:47.565563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:47.565514 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:09:47.565755 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:47.565588 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:09:48.399986 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:09:48.399896 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:09:48.566031 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:48.565444 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:09:48.566031 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:48.565538 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:09:48.606481 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:48.606431 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:09:48.606698 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:48.606491 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:09:49.468542 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:49.468511 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:49.468983 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:49.468959 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:49.469898 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:49.469874 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" Nov 03 23:09:49.647829 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:49.647454 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:09:49.647829 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:49.647528 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:09:49.696075 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:49.696038 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:49.696482 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:49.696460 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:49.697161 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:49.697142 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" Nov 03 23:09:55.032282 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:55.032234 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr"] Nov 03 23:09:55.042416 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:55.042385 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a23ddf01-7p7jr"] Nov 03 23:09:55.731393 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:55.731358 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2709af96-abe6-4467-b9ad-e32c7dd469aa" path="/var/lib/kubelet/pods/2709af96-abe6-4467-b9ad-e32c7dd469aa/volumes" Nov 03 23:09:56.026918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:56.026833 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq"] Nov 03 23:09:56.040865 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:56.040822 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf6edce4-g5srq"] Nov 03 23:09:57.732039 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:57.732000 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf36ae2-a437-4a1e-adc5-b554beb18795" path="/var/lib/kubelet/pods/0bf36ae2-a437-4a1e-adc5-b554beb18795/volumes" Nov 03 23:09:58.392861 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:09:58.392824 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:10:00.728858 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:00.728823 4903 scope.go:122] "RemoveContainer" containerID="b22ec8e2c20b5a7b127536297fbc3aaa0f4b121afa8562c12251ccfffc7ff4fe" Nov 03 23:10:03.025955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:03.025920 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz"] Nov 03 23:10:03.036088 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:03.036057 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a359e6-dsvqz"] Nov 03 23:10:03.731959 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:03.731923 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e9ee770-79c3-4638-ab08-fc36e583b704" path="/var/lib/kubelet/pods/2e9ee770-79c3-4638-ab08-fc36e583b704/volumes" Nov 03 23:10:14.026563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:14.026526 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8"] Nov 03 23:10:14.036889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:14.036859 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14689fd3-x8lk8"] Nov 03 23:10:15.731934 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:15.731897 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefb864d-6f1a-418b-b77e-7163001cfd19" path="/var/lib/kubelet/pods/cefb864d-6f1a-418b-b77e-7163001cfd19/volumes" Nov 03 23:10:18.382387 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:10:18.382351 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:10:21.719788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:21.719746 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:10:32.172476 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:32.172440 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:10:37.209329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:37.209223 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:10:46.802309 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:46.802252 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m"] Nov 03 23:10:46.802732 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:46.802471 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" podUID="e74fd6ab-74b5-4b05-b339-6c20da28e431" containerName="discovery" containerID="cri-o://4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502" gracePeriod=30 Nov 03 23:10:47.062310 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.062241 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:10:47.096656 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.096626 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-token\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-token\") pod \"e74fd6ab-74b5-4b05-b339-6c20da28e431\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " Nov 03 23:10:47.096802 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.096665 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74fd6ab-74b5-4b05-b339-6c20da28e431-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74fd6ab-74b5-4b05-b339-6c20da28e431-local-certs\") pod \"e74fd6ab-74b5-4b05-b339-6c20da28e431\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " Nov 03 23:10:47.096802 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.096743 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-cacerts\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-cacerts\") pod \"e74fd6ab-74b5-4b05-b339-6c20da28e431\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " Nov 03 23:10:47.096802 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.096772 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-kube-api-access-fthll\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-kube-api-access-fthll\") pod \"e74fd6ab-74b5-4b05-b339-6c20da28e431\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " Nov 03 23:10:47.096964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.096815 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-ca-configmap\") pod \"e74fd6ab-74b5-4b05-b339-6c20da28e431\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " Nov 03 23:10:47.096964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.096839 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-dns-cert\") pod \"e74fd6ab-74b5-4b05-b339-6c20da28e431\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " Nov 03 23:10:47.096964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.096859 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-kubeconfig\") pod \"e74fd6ab-74b5-4b05-b339-6c20da28e431\" (UID: \"e74fd6ab-74b5-4b05-b339-6c20da28e431\") " Nov 03 23:10:47.097329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.097270 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-ca-configmap" pod "e74fd6ab-74b5-4b05-b339-6c20da28e431" (UID: "e74fd6ab-74b5-4b05-b339-6c20da28e431"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:10:47.099160 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.099102 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74fd6ab-74b5-4b05-b339-6c20da28e431-local-certs" pod "e74fd6ab-74b5-4b05-b339-6c20da28e431" (UID: "e74fd6ab-74b5-4b05-b339-6c20da28e431"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:10:47.099160 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.099148 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-token" pod "e74fd6ab-74b5-4b05-b339-6c20da28e431" (UID: "e74fd6ab-74b5-4b05-b339-6c20da28e431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:47.099374 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.099259 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-cacerts" pod "e74fd6ab-74b5-4b05-b339-6c20da28e431" (UID: "e74fd6ab-74b5-4b05-b339-6c20da28e431"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:47.099451 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.099397 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-dns-cert" pod "e74fd6ab-74b5-4b05-b339-6c20da28e431" (UID: "e74fd6ab-74b5-4b05-b339-6c20da28e431"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:47.099512 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.099468 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-kube-api-access-fthll" pod "e74fd6ab-74b5-4b05-b339-6c20da28e431" (UID: "e74fd6ab-74b5-4b05-b339-6c20da28e431"). InnerVolumeSpecName "kube-api-access-fthll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:10:47.099512 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.099470 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-kubeconfig" pod "e74fd6ab-74b5-4b05-b339-6c20da28e431" (UID: "e74fd6ab-74b5-4b05-b339-6c20da28e431"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:10:47.197629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.197592 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:10:47.197629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.197624 4903 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74fd6ab-74b5-4b05-b339-6c20da28e431-local-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:10:47.197629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.197634 4903 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-cacerts\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:10:47.197890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.197645 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fthll\" (UniqueName: \"kubernetes.io/projected/e74fd6ab-74b5-4b05-b339-6c20da28e431-kube-api-access-fthll\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:10:47.197890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.197655 4903 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-ca-configmap\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:10:47.197890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.197664 4903 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-csr-dns-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:10:47.197890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.197672 4903 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e74fd6ab-74b5-4b05-b339-6c20da28e431-istio-kubeconfig\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:10:47.773638 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.773605 4903 generic.go:356] "Generic (PLEG): container finished" podID="e74fd6ab-74b5-4b05-b339-6c20da28e431" containerID="4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502" exitCode=0 Nov 03 23:10:47.773825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.773645 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" event={"ID":"e74fd6ab-74b5-4b05-b339-6c20da28e431","Type":"ContainerDied","Data":"4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502"} Nov 03 23:10:47.773825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.773683 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" Nov 03 23:10:47.773825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.773716 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m" event={"ID":"e74fd6ab-74b5-4b05-b339-6c20da28e431","Type":"ContainerDied","Data":"8ead431be2ca59cd568584123ac708a4dee55c4e867def4cdc1598b024f4604b"} Nov 03 23:10:47.773825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.773741 4903 scope.go:122] "RemoveContainer" containerID="4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502" Nov 03 23:10:47.784221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.784201 4903 scope.go:122] "RemoveContainer" containerID="4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502" Nov 03 23:10:47.784505 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:10:47.784486 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502\": container with ID starting with 4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502 not found: ID does not exist" containerID="4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502" Nov 03 23:10:47.784585 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.784526 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502"} err="failed to get container status \"4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502\": rpc error: code = NotFound desc = could not find container \"4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502\": container with ID starting with 4f44b4c8bf36d0740087fc3d184b9ea070521f3ba8108069eeb228d596375502 not found: ID does not exist" Nov 03 23:10:47.830207 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.830171 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m"] Nov 03 23:10:47.840286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:47.840260 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mzm2m"] Nov 03 23:10:48.382561 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:10:48.382528 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:10:49.731842 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:49.731806 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74fd6ab-74b5-4b05-b339-6c20da28e431" path="/var/lib/kubelet/pods/e74fd6ab-74b5-4b05-b339-6c20da28e431/volumes" Nov 03 23:10:53.803598 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:53.803565 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:10:59.122842 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:10:59.122802 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:11:00.814195 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:00.814160 4903 scope.go:122] "RemoveContainer" containerID="84f67e14d4cba0f87290d252265eb26e3ce6c85c58da1b54e17b5a10df595b49" Nov 03 23:11:00.823594 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:00.823571 4903 scope.go:122] "RemoveContainer" containerID="3d9edf6ac45a4b3851ba403eea027f4ab1568100a016860c167f4f7001a72f63" Nov 03 23:11:00.832354 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:00.832334 4903 scope.go:122] "RemoveContainer" containerID="e4131cc25c8a90f8a348f9285555f895471557f54bbea51434b586b6479d42f6" Nov 03 23:11:00.841303 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:00.841267 4903 scope.go:122] "RemoveContainer" containerID="f02de15c74272a309ea7c7843965c6498d8253dc594835a087864c9d51a3ab02" Nov 03 23:11:00.849720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:00.849698 4903 scope.go:122] "RemoveContainer" containerID="3372923062b5db0a8c2a55123f5d5dcf9bce6e202b40a657a04a6833e5b5f680" Nov 03 23:11:00.858826 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:00.858805 4903 scope.go:122] "RemoveContainer" containerID="9425ff9bf5af3438227a94e6f671b24527d1c1d64f1d957b0e370c145a6d53ca" Nov 03 23:11:00.867372 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:00.867348 4903 scope.go:122] "RemoveContainer" containerID="e9cea010680709d13583251d76561e992474498b83ad4ad34df489c4accf90e0" Nov 03 23:11:00.875575 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:00.875548 4903 scope.go:122] "RemoveContainer" containerID="2b9c6cf78db7b802b10e28808cf527723885ce02122d9b4cc0b6847138fd3e07" Nov 03 23:11:04.027077 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:04.027042 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp"] Nov 03 23:11:04.037237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:04.037194 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce90165-vvgkp"] Nov 03 23:11:05.732627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:05.732589 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6edc41c7-1c97-4f3d-98b1-5182d9be168b" path="/var/lib/kubelet/pods/6edc41c7-1c97-4f3d-98b1-5182d9be168b/volumes" Nov 03 23:11:15.028025 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:15.027989 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w"] Nov 03 23:11:15.037983 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:15.037956 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc728923-qqh6w"] Nov 03 23:11:15.731984 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:15.731944 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0a0428-aefd-41d0-abb1-0b211b671d92" path="/var/lib/kubelet/pods/6e0a0428-aefd-41d0-abb1-0b211b671d92/volumes" Nov 03 23:11:16.195909 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:16.195876 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:11:18.383057 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:11:18.383010 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:11:21.502264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:21.502227 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:11:24.026484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:24.026450 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst"] Nov 03 23:11:24.036508 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:24.036482 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9"] Nov 03 23:11:24.038030 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:24.038009 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cd63b87-dmxst"] Nov 03 23:11:24.039405 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:24.039382 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62e2c3cb-kxxl9"] Nov 03 23:11:25.731393 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:25.731358 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08eded23-13b5-438a-b832-30b199e9984c" path="/var/lib/kubelet/pods/08eded23-13b5-438a-b832-30b199e9984c/volumes" Nov 03 23:11:25.731891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:25.731791 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5" path="/var/lib/kubelet/pods/c58e8ec9-0464-4e28-9bf1-bb6d8af30ae5/volumes" Nov 03 23:11:29.025465 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:29.025432 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm"] Nov 03 23:11:29.035586 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:29.035561 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4986d175-7cgkm"] Nov 03 23:11:29.732608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:29.732563 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a6a9ff7-6e2a-420d-bc42-847c128a1b6c" path="/var/lib/kubelet/pods/3a6a9ff7-6e2a-420d-bc42-847c128a1b6c/volumes" Nov 03 23:11:30.026618 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:30.026535 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw"] Nov 03 23:11:30.037017 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:30.036985 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be1b8fc6-2grgw"] Nov 03 23:11:31.732312 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:31.732257 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4017b29f-6cd4-484e-a0be-e97243142e4f" path="/var/lib/kubelet/pods/4017b29f-6cd4-484e-a0be-e97243142e4f/volumes" Nov 03 23:11:38.315308 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:38.315263 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:11:39.028710 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:39.028671 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt"] Nov 03 23:11:39.039434 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:39.039389 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61aa25dc-vx5xt"] Nov 03 23:11:39.041742 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:39.041706 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj"] Nov 03 23:11:39.043856 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:39.043831 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cf82f1-w6mvj"] Nov 03 23:11:39.733180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:39.733148 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c4c43b-3b7e-4b8b-8a0f-b813d655c987" path="/var/lib/kubelet/pods/07c4c43b-3b7e-4b8b-8a0f-b813d655c987/volumes" Nov 03 23:11:39.734111 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:39.734080 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e10c32-f03d-4b47-9b73-c04087642e9b" path="/var/lib/kubelet/pods/65e10c32-f03d-4b47-9b73-c04087642e9b/volumes" Nov 03 23:11:48.383527 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:11:48.383489 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:11:54.385968 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:11:54.385930 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:12:00.935333 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:00.935284 4903 scope.go:122] "RemoveContainer" containerID="79abe1ef67277dd83751d0fcc25a67a762155c7762e1a73ba7a32fc38165cbb8" Nov 03 23:12:00.944462 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:00.944443 4903 scope.go:122] "RemoveContainer" containerID="0acb52f053ff7d3cc13baf809d37e4bb79e523878b268895509d9b8153ced0e9" Nov 03 23:12:00.952675 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:00.952656 4903 scope.go:122] "RemoveContainer" containerID="d209c0abd8ff213078e0bb031cccfe9f210459780cbbd0d7c92e289f4f8fefba" Nov 03 23:12:00.960607 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:00.960586 4903 scope.go:122] "RemoveContainer" containerID="78e6669fdb5b4e5070082243b21f7418915d15e3b4a6e820898ef4e9008cb426" Nov 03 23:12:00.968484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:00.968462 4903 scope.go:122] "RemoveContainer" containerID="4ae2f3d22da3fb86945d1f6720a1d358a89703764ac0a656d65ef666587bb996" Nov 03 23:12:00.976179 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:00.976153 4903 scope.go:122] "RemoveContainer" containerID="04088a0cef670217266f36b9574d4e74e92344e1da9a6c61ff8192c253d3f283" Nov 03 23:12:00.983919 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:00.983890 4903 scope.go:122] "RemoveContainer" containerID="d66d37842e91fbb1a73d26d9ddec6c66abe94109585a560554f1d35f12224eae" Nov 03 23:12:00.991641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:00.991622 4903 scope.go:122] "RemoveContainer" containerID="3d9ff547a3b1ccf365b1ce53259fed772cfba9465a2aca4f5b982e25bb793e1a" Nov 03 23:12:00.999095 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:00.999070 4903 scope.go:122] "RemoveContainer" containerID="8a3fc8812e213d11baa410fcca57840e9eb94beab799a9568f3ef3455a08d28c" Nov 03 23:12:05.211047 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:05.211014 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:12:09.892370 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:09.889329 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:12:12.030988 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:12.030949 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj"] Nov 03 23:12:12.041818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:12.041785 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6db3168-67xzj"] Nov 03 23:12:13.730973 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:13.730937 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a101b815-7b9a-4f8a-9d5c-31b3c4c4defd" path="/var/lib/kubelet/pods/a101b815-7b9a-4f8a-9d5c-31b3c4c4defd/volumes" Nov 03 23:12:18.383710 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:12:18.383663 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:12:26.283168 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:26.283132 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:12:28.026883 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:28.026846 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh"] Nov 03 23:12:28.046205 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:28.038346 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73e709c5-gwfzh"] Nov 03 23:12:29.734223 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:29.734168 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d987314-2b17-44cc-be49-142bcf987e1c" path="/var/lib/kubelet/pods/5d987314-2b17-44cc-be49-142bcf987e1c/volumes" Nov 03 23:12:31.570726 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:31.570688 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:12:41.471064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:41.471027 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:12:48.382897 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:12:48.382852 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:12:54.463975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:54.463945 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:12:54.471071 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:54.471046 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:12:54.472043 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:54.472022 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:12:54.478667 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:54.478649 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:12:59.287878 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:12:59.287839 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:13:01.087319 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.078032 4903 scope.go:122] "RemoveContainer" containerID="f2e7d9cf32ae46b07b4c66173d38040c39a81a85c42f7d4c883056a70e757cc0" Nov 03 23:13:01.118382 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.118360 4903 scope.go:122] "RemoveContainer" containerID="5002fcaac59c7c81d1ef7fec14e53bef86852a85a4ceb27db905ba8e7f7e7059" Nov 03 23:13:01.127377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.127357 4903 scope.go:122] "RemoveContainer" containerID="963f27dbbf3fa1c91e8ad0f5b54b93b111611fc5da2fbbc6d387528d41727a5a" Nov 03 23:13:01.135180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.135158 4903 scope.go:122] "RemoveContainer" containerID="8601f34efd45cfc5a14c94449a501bc8cf3e57b76180849ea20727c13950b7fb" Nov 03 23:13:01.198314 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.194579 4903 scope.go:122] "RemoveContainer" containerID="4bfe2d0f2daf70fe01c204f539cd1ff1749ec29f20b1347a74cd32507c1e81b8" Nov 03 23:13:01.222186 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.222160 4903 scope.go:122] "RemoveContainer" containerID="06b80075c4bd7a60ef7ea6e4707649a6ea3065df5cc468c1c3ad27a5da46b576" Nov 03 23:13:01.230106 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.230086 4903 scope.go:122] "RemoveContainer" containerID="0e815766d0407b278c8eacec8491005575321c55b04bbe08031a3609095edd05" Nov 03 23:13:01.237675 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.237658 4903 scope.go:122] "RemoveContainer" containerID="98b9e306ef271b124efdd47a2d2c93567823edf3e333cbdef9704223e6c4e3c6" Nov 03 23:13:01.249507 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.249482 4903 scope.go:122] "RemoveContainer" containerID="f920f84dab83a0ed132c72ae5e0c24222666d754a51d325d9f58b4a909bcfbfd" Nov 03 23:13:01.268966 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.268705 4903 scope.go:122] "RemoveContainer" containerID="9836b349d7265c6632c041286c57d7099ee1bb4790b8e0605b1c800def897ec5" Nov 03 23:13:01.318312 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.318256 4903 scope.go:122] "RemoveContainer" containerID="a2ea66dbfdd72868dcf9767709d38b253595f06d4c3033a3206c4cc9beaaee6c" Nov 03 23:13:01.330216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.330192 4903 scope.go:122] "RemoveContainer" containerID="aaa10baf45b65c8ffc5d64785419432ed1b94fe6401335d329ff3521c7078647" Nov 03 23:13:01.338211 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.338191 4903 scope.go:122] "RemoveContainer" containerID="ca0eb418e76bd56f17845a2c394384ced158ceff128e43084db1bcc52c4f82c0" Nov 03 23:13:01.412717 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.409474 4903 scope.go:122] "RemoveContainer" containerID="d1ca1478b4323b5f4a7d355e87a7720f46e15f1e2ea009713e71a3095c0b695b" Nov 03 23:13:01.426497 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:01.423916 4903 scope.go:122] "RemoveContainer" containerID="0a68921f768c5ae999603c7784d59b03315628b7bb51cec0d22b49632e87b297" Nov 03 23:13:09.592210 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:09.592170 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:13:18.382232 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:13:18.382197 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:13:29.600892 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:29.600852 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:13:35.402023 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:35.401988 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:13:39.389183 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.389073 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4"] Nov 03 23:13:39.389784 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.389563 4903 cpu_manager.go:414] "containerMap: removing container" podUID="e74fd6ab-74b5-4b05-b339-6c20da28e431" containerName="discovery" Nov 03 23:13:39.389784 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.389578 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74fd6ab-74b5-4b05-b339-6c20da28e431" containerName="discovery" Nov 03 23:13:39.389784 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.389666 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74fd6ab-74b5-4b05-b339-6c20da28e431" containerName="discovery" Nov 03 23:13:39.393051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.393025 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.407763 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.407728 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d5590b-openshift-default-dockercfg-hfqnm\"" Nov 03 23:13:39.419994 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.419960 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4"] Nov 03 23:13:39.493662 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.493610 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-envoy\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.493871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.493668 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-data\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.493871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.493757 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-socket\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.493871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.493817 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ecf6f8-6379-41c5-9d51-b30162893bc8-istiod-ca-cert\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.494068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.493894 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-token\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.494068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.493942 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-credential-socket\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.494068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.493981 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-certs\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.494068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.494001 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsb82\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-kube-api-access-xsb82\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.494068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.494032 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-podinfo\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.595447 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.595401 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-socket\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.595670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.595546 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ecf6f8-6379-41c5-9d51-b30162893bc8-istiod-ca-cert\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.595670 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.595658 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-token\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.595801 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.595725 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-credential-socket\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.595801 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.595779 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-certs\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.595927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.595810 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsb82\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-kube-api-access-xsb82\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.595927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.595902 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-socket\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.596041 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.595973 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-podinfo\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.596104 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.596049 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-envoy\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.596104 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.596095 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-data\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.596220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.596137 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-credential-socket\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.596282 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.596251 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-certs\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.596453 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.596435 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-data\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.597136 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.597101 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ecf6f8-6379-41c5-9d51-b30162893bc8-istiod-ca-cert\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.598871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.598827 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-envoy\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.599268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.599246 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-podinfo\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.617396 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.617368 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-token\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.617525 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.617461 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsb82\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-kube-api-access-xsb82\") pod \"e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:39.706511 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:39.706480 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:40.022222 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:40.022138 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4"] Nov 03 23:13:40.025454 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:13:40.025427 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ecf6f8_6379_41c5_9d51_b30162893bc8.slice/crio-fce15310b270fdeea654dde09bd6babd4eff1377c8915bc2a9f259207f96b7fe WatchSource:0}: Error finding container fce15310b270fdeea654dde09bd6babd4eff1377c8915bc2a9f259207f96b7fe: Status 404 returned error can't find the container with id fce15310b270fdeea654dde09bd6babd4eff1377c8915bc2a9f259207f96b7fe Nov 03 23:13:40.027622 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:40.027591 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:13:40.027696 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:40.027659 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:13:40.027696 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:40.027687 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:13:40.915042 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:40.914996 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" event={"ID":"43ecf6f8-6379-41c5-9d51-b30162893bc8","Type":"ContainerStarted","Data":"22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9"} Nov 03 23:13:40.915568 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:40.915054 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" event={"ID":"43ecf6f8-6379-41c5-9d51-b30162893bc8","Type":"ContainerStarted","Data":"fce15310b270fdeea654dde09bd6babd4eff1377c8915bc2a9f259207f96b7fe"} Nov 03 23:13:40.966877 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:40.966825 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" podStartSLOduration=1.966811069 podStartE2EDuration="1.966811069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:13:40.965962684 +0000 UTC m=+3647.369903661" watchObservedRunningTime="2026-11-03 23:13:40.966811069 +0000 UTC m=+3647.370752044" Nov 03 23:13:41.707083 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:41.707048 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:41.712601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:41.712574 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:41.919891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:41.919856 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:41.920808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:41.920789 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:13:45.299232 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.299195 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8"] Nov 03 23:13:45.302979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.302961 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:45.329515 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.329486 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d5590b\"" Nov 03 23:13:45.329711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.329573 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4582222a-dockercfg-mfvt8\"" Nov 03 23:13:45.331050 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.331025 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8"] Nov 03 23:13:45.348818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.348778 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtphg\" (UniqueName: \"kubernetes.io/projected/1bb8be41-533c-4d1d-af6b-c2d636f8db66-kube-api-access-jtphg\") pod \"payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8\" (UID: \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:45.348953 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.348930 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb8be41-533c-4d1d-af6b-c2d636f8db66-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8\" (UID: \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:45.449952 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.449914 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtphg\" (UniqueName: \"kubernetes.io/projected/1bb8be41-533c-4d1d-af6b-c2d636f8db66-kube-api-access-jtphg\") pod \"payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8\" (UID: \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:45.450119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.449983 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb8be41-533c-4d1d-af6b-c2d636f8db66-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8\" (UID: \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:45.450681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.450659 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb8be41-533c-4d1d-af6b-c2d636f8db66-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8\" (UID: \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:45.473509 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.473482 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtphg\" (UniqueName: \"kubernetes.io/projected/1bb8be41-533c-4d1d-af6b-c2d636f8db66-kube-api-access-jtphg\") pod \"payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8\" (UID: \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:45.616068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.615985 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:45.915770 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.915737 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8"] Nov 03 23:13:45.918817 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:13:45.918790 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb8be41_533c_4d1d_af6b_c2d636f8db66.slice/crio-6f0dba651bb1cc07fb7ad973becc163c022173f30fe4662011c4bab2073266d5 WatchSource:0}: Error finding container 6f0dba651bb1cc07fb7ad973becc163c022173f30fe4662011c4bab2073266d5: Status 404 returned error can't find the container with id 6f0dba651bb1cc07fb7ad973becc163c022173f30fe4662011c4bab2073266d5 Nov 03 23:13:45.938094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:45.938064 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" event={"ID":"1bb8be41-533c-4d1d-af6b-c2d636f8db66","Type":"ContainerStarted","Data":"6f0dba651bb1cc07fb7ad973becc163c022173f30fe4662011c4bab2073266d5"} Nov 03 23:13:46.943784 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:46.943744 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" event={"ID":"1bb8be41-533c-4d1d-af6b-c2d636f8db66","Type":"ContainerStarted","Data":"1ae5c706899471c8ca4aef4e77c02ea809aed2f38c7bb8352e47133d93c901c5"} Nov 03 23:13:46.944224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:46.943807 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:46.982110 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:46.982058 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" podStartSLOduration=1.982045121 podStartE2EDuration="1.982045121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:13:46.98045518 +0000 UTC m=+3653.384396169" watchObservedRunningTime="2026-11-03 23:13:46.982045121 +0000 UTC m=+3653.385986096" Nov 03 23:13:48.384314 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:13:48.384249 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:13:48.915985 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:48.915947 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:13:57.951769 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:57.951736 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:13:57.961440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:57.961415 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:13:57.961565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:57.961461 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:13:59.020073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:59.020035 4903 generic.go:356] "Generic (PLEG): container finished" podID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerID="22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9" exitCode=0 Nov 03 23:13:59.020558 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:59.020103 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" event={"ID":"43ecf6f8-6379-41c5-9d51-b30162893bc8","Type":"ContainerDied","Data":"22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9"} Nov 03 23:13:59.020558 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:59.020471 4903 scope.go:122] "RemoveContainer" containerID="22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9" Nov 03 23:13:59.021568 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:59.021541 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:13:59.021634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:59.021610 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:13:59.021675 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:13:59.021642 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:14:00.027080 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.027041 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" event={"ID":"43ecf6f8-6379-41c5-9d51-b30162893bc8","Type":"ContainerStarted","Data":"b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25"} Nov 03 23:14:00.227307 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.227269 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn"] Nov 03 23:14:00.229881 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.229865 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.252325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.251281 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn"] Nov 03 23:14:00.252325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.251516 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d5590b\"/\"default-dockercfg-tx2m6\"" Nov 03 23:14:00.252325 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.251696 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d5590b\"/\"simulator-single-main-d5590b-kserve-self-signed-certs\"" Nov 03 23:14:00.305227 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.305135 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d5590b\"/\"kube-root-ca.crt\"" Nov 03 23:14:00.305227 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.305184 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d5590b\"/\"openshift-service-ca.crt\"" Nov 03 23:14:00.325708 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.325678 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xsqp\" (UniqueName: \"kubernetes.io/projected/40bae792-cb9d-49de-8666-b23ae8db65f5-kube-api-access-5xsqp\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.325839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.325714 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-home\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.325839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.325731 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-model-cache\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.325918 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.325877 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-dshm\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.325962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.325944 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bae792-cb9d-49de-8666-b23ae8db65f5-tls-certs\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.326005 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.325990 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-tmp-dir\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.427065 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.427029 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-dshm\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.427249 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.427095 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bae792-cb9d-49de-8666-b23ae8db65f5-tls-certs\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.427249 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.427119 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-tmp-dir\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.427249 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.427171 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xsqp\" (UniqueName: \"kubernetes.io/projected/40bae792-cb9d-49de-8666-b23ae8db65f5-kube-api-access-5xsqp\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.427249 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.427192 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-home\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.427444 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.427245 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-model-cache\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.427632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.427603 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-home\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.427632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.427626 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-model-cache\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.427819 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.427653 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-tmp-dir\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.429478 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.429450 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-dshm\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.429729 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.429713 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bae792-cb9d-49de-8666-b23ae8db65f5-tls-certs\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.461328 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.461283 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xsqp\" (UniqueName: \"kubernetes.io/projected/40bae792-cb9d-49de-8666-b23ae8db65f5-kube-api-access-5xsqp\") pod \"simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.548563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.548525 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:00.707480 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.707440 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:14:00.707729 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.707702 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 23:14:00.707820 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.707761 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 23:14:00.867723 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:00.867694 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn"] Nov 03 23:14:00.869718 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:14:00.869690 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40bae792_cb9d_49de_8666_b23ae8db65f5.slice/crio-ee430e37d425714be289a76f19997c8ac06f821c07f211b2c6ee39c5a8868291 WatchSource:0}: Error finding container ee430e37d425714be289a76f19997c8ac06f821c07f211b2c6ee39c5a8868291: Status 404 returned error can't find the container with id ee430e37d425714be289a76f19997c8ac06f821c07f211b2c6ee39c5a8868291 Nov 03 23:14:01.035175 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:01.035134 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" event={"ID":"40bae792-cb9d-49de-8666-b23ae8db65f5","Type":"ContainerStarted","Data":"6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72"} Nov 03 23:14:01.035623 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:01.035189 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" event={"ID":"40bae792-cb9d-49de-8666-b23ae8db65f5","Type":"ContainerStarted","Data":"ee430e37d425714be289a76f19997c8ac06f821c07f211b2c6ee39c5a8868291"} Nov 03 23:14:01.085897 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:01.085840 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" podStartSLOduration=1.085825671 podStartE2EDuration="1.085825671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:14:01.083765219 +0000 UTC m=+3667.487706193" watchObservedRunningTime="2026-11-03 23:14:01.085825671 +0000 UTC m=+3667.489766636" Nov 03 23:14:01.565321 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:01.552627 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:01.663319 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:01.654746 4903 scope.go:122] "RemoveContainer" containerID="e33640548b66bf87c4b433fad514717feb80757ca888bb9c6148ecc2350e77a1" Nov 03 23:14:01.674228 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:01.674200 4903 scope.go:122] "RemoveContainer" containerID="bba5697ed3aa8e78ea422f40add58016e9998686101e46bdea44fedf17029cc8" Nov 03 23:14:01.684754 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:01.684729 4903 scope.go:122] "RemoveContainer" containerID="1582dc0de8f8832c65ab7de18e873996f1ba9d287f611e0cff2098072b56e725" Nov 03 23:14:01.706855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:01.706824 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 23:14:01.706953 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:01.706886 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 23:14:02.575901 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:02.575857 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:14:02.707611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:02.707569 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 23:14:02.707787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:02.707646 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 23:14:03.706984 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:03.706947 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 23:14:03.707420 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:03.706999 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 23:14:04.707784 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:04.707747 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 23:14:04.708192 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:04.707813 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 23:14:05.711065 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:05.711034 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:14:05.711606 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:05.711327 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:14:05.711997 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:05.711980 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:14:10.549875 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:10.549841 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:10.562118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:10.562093 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:11.115711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:11.115676 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:14:18.384407 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:14:18.384364 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:14:20.002385 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:20.002348 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:14:32.523064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.523029 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2"] Nov 03 23:14:32.526139 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.526112 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.541894 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.541869 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:14:32.552110 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.552088 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:14:32.552110 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.552089 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:14:32.552495 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.552147 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:14:32.553980 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.553963 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2"] Nov 03 23:14:32.656194 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.656163 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6gh\" (UniqueName: \"kubernetes.io/projected/fbfcc126-780f-4692-a0d9-c4b18197d1eb-kube-api-access-2h6gh\") pod \"maas-api-revoke-sub-07cbd2ec-bpxc2\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.656416 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.656218 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07cbd2ec-bpxc2\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.656475 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.656409 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07cbd2ec-bpxc2\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.756881 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.756853 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07cbd2ec-bpxc2\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.757068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.756916 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6gh\" (UniqueName: \"kubernetes.io/projected/fbfcc126-780f-4692-a0d9-c4b18197d1eb-kube-api-access-2h6gh\") pod \"maas-api-revoke-sub-07cbd2ec-bpxc2\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.757068 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.756951 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07cbd2ec-bpxc2\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.757599 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.757578 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07cbd2ec-bpxc2\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.759346 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.759329 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07cbd2ec-bpxc2\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.778329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.778245 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6gh\" (UniqueName: \"kubernetes.io/projected/fbfcc126-780f-4692-a0d9-c4b18197d1eb-kube-api-access-2h6gh\") pod \"maas-api-revoke-sub-07cbd2ec-bpxc2\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:32.838832 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:32.838798 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:33.135436 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:33.135406 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2"] Nov 03 23:14:33.138028 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:14:33.137993 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfcc126_780f_4692_a0d9_c4b18197d1eb.slice/crio-8192709ca71c7e784d6256119a36c93db42221e2488ee9a4a6ef67ad99d9d904 WatchSource:0}: Error finding container 8192709ca71c7e784d6256119a36c93db42221e2488ee9a4a6ef67ad99d9d904: Status 404 returned error can't find the container with id 8192709ca71c7e784d6256119a36c93db42221e2488ee9a4a6ef67ad99d9d904 Nov 03 23:14:33.210927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:33.210891 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" event={"ID":"fbfcc126-780f-4692-a0d9-c4b18197d1eb","Type":"ContainerStarted","Data":"8192709ca71c7e784d6256119a36c93db42221e2488ee9a4a6ef67ad99d9d904"} Nov 03 23:14:34.217135 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:34.217094 4903 generic.go:356] "Generic (PLEG): container finished" podID="fbfcc126-780f-4692-a0d9-c4b18197d1eb" containerID="cda66b0e1199001af0d0221574b33deeabeafed0132bb9e501f2e2ae3633fbc5" exitCode=0 Nov 03 23:14:34.217630 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:34.217184 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" event={"ID":"fbfcc126-780f-4692-a0d9-c4b18197d1eb","Type":"ContainerDied","Data":"cda66b0e1199001af0d0221574b33deeabeafed0132bb9e501f2e2ae3633fbc5"} Nov 03 23:14:35.362077 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.362052 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:35.483572 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.483494 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfcc126-780f-4692-a0d9-c4b18197d1eb-kube-api-access-2h6gh\" (UniqueName: \"kubernetes.io/projected/fbfcc126-780f-4692-a0d9-c4b18197d1eb-kube-api-access-2h6gh\") pod \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " Nov 03 23:14:35.483572 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.483533 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-xks-ca\") pod \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " Nov 03 23:14:35.483572 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.483565 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-openshift-ca\") pod \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\" (UID: \"fbfcc126-780f-4692-a0d9-c4b18197d1eb\") " Nov 03 23:14:35.483983 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.483957 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-openshift-ca" pod "fbfcc126-780f-4692-a0d9-c4b18197d1eb" (UID: "fbfcc126-780f-4692-a0d9-c4b18197d1eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:35.485663 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.485640 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfcc126-780f-4692-a0d9-c4b18197d1eb-kube-api-access-2h6gh" pod "fbfcc126-780f-4692-a0d9-c4b18197d1eb" (UID: "fbfcc126-780f-4692-a0d9-c4b18197d1eb"). InnerVolumeSpecName "kube-api-access-2h6gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:35.485757 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.485655 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-xks-ca" pod "fbfcc126-780f-4692-a0d9-c4b18197d1eb" (UID: "fbfcc126-780f-4692-a0d9-c4b18197d1eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:35.584608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.584573 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h6gh\" (UniqueName: \"kubernetes.io/projected/fbfcc126-780f-4692-a0d9-c4b18197d1eb-kube-api-access-2h6gh\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:14:35.584608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.584606 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:14:35.584811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:35.584619 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcc126-780f-4692-a0d9-c4b18197d1eb-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:14:36.227387 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:36.227351 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" Nov 03 23:14:36.227387 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:36.227366 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2" event={"ID":"fbfcc126-780f-4692-a0d9-c4b18197d1eb","Type":"ContainerDied","Data":"8192709ca71c7e784d6256119a36c93db42221e2488ee9a4a6ef67ad99d9d904"} Nov 03 23:14:36.227629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:36.227409 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8192709ca71c7e784d6256119a36c93db42221e2488ee9a4a6ef67ad99d9d904" Nov 03 23:14:43.909988 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.909951 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd"] Nov 03 23:14:43.910413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.910357 4903 cpu_manager.go:414] "containerMap: removing container" podUID="fbfcc126-780f-4692-a0d9-c4b18197d1eb" containerName="revoke-keys" Nov 03 23:14:43.910413 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.910368 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfcc126-780f-4692-a0d9-c4b18197d1eb" containerName="revoke-keys" Nov 03 23:14:43.910504 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.910451 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfcc126-780f-4692-a0d9-c4b18197d1eb" containerName="revoke-keys" Nov 03 23:14:43.912621 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.912604 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:43.925627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.925595 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:14:43.925627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.925610 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:14:43.925627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.925600 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:14:43.925836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.925631 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:14:43.931147 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:43.931123 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd"] Nov 03 23:14:44.067130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.067097 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4bm9\" (UniqueName: \"kubernetes.io/projected/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-kube-api-access-z4bm9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.067337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.067211 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.067337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.067275 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.168539 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.168444 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4bm9\" (UniqueName: \"kubernetes.io/projected/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-kube-api-access-z4bm9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.168539 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.168540 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.168802 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.168587 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.169206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.169169 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.170964 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.170942 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.189476 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.189456 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4bm9\" (UniqueName: \"kubernetes.io/projected/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-kube-api-access-z4bm9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.224392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.224359 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:44.519273 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:44.519199 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd"] Nov 03 23:14:44.522373 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:14:44.522346 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cc9eda_54ea_49fc_80c7_b3138b7abb7b.slice/crio-93b49b3ed685600d5f0749111edeeda2d156644228b8830cb62ec2da9118ecff WatchSource:0}: Error finding container 93b49b3ed685600d5f0749111edeeda2d156644228b8830cb62ec2da9118ecff: Status 404 returned error can't find the container with id 93b49b3ed685600d5f0749111edeeda2d156644228b8830cb62ec2da9118ecff Nov 03 23:14:45.075704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:45.075673 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:14:45.271696 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:45.271654 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" event={"ID":"44cc9eda-54ea-49fc-80c7-b3138b7abb7b","Type":"ContainerStarted","Data":"199b00332a05deeb13f59788edf28bb0f3da9c5c67d32f3ecfb75715066b28c4"} Nov 03 23:14:45.271892 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:45.271713 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" event={"ID":"44cc9eda-54ea-49fc-80c7-b3138b7abb7b","Type":"ContainerStarted","Data":"93b49b3ed685600d5f0749111edeeda2d156644228b8830cb62ec2da9118ecff"} Nov 03 23:14:45.312818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:45.312749 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" podStartSLOduration=2.312734151 podStartE2EDuration="2.312734151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:14:45.308934797 +0000 UTC m=+3711.712875774" watchObservedRunningTime="2026-11-03 23:14:45.312734151 +0000 UTC m=+3711.716675162" Nov 03 23:14:46.278286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:46.278244 4903 generic.go:356] "Generic (PLEG): container finished" podID="44cc9eda-54ea-49fc-80c7-b3138b7abb7b" containerID="199b00332a05deeb13f59788edf28bb0f3da9c5c67d32f3ecfb75715066b28c4" exitCode=0 Nov 03 23:14:46.278758 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:46.278345 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" event={"ID":"44cc9eda-54ea-49fc-80c7-b3138b7abb7b","Type":"ContainerDied","Data":"199b00332a05deeb13f59788edf28bb0f3da9c5c67d32f3ecfb75715066b28c4"} Nov 03 23:14:47.419500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.419471 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:47.603274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.603188 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-kube-api-access-z4bm9\" (UniqueName: \"kubernetes.io/projected/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-kube-api-access-z4bm9\") pod \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " Nov 03 23:14:47.603274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.603255 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-openshift-ca\") pod \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " Nov 03 23:14:47.603492 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.603361 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-xks-ca\") pod \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\" (UID: \"44cc9eda-54ea-49fc-80c7-b3138b7abb7b\") " Nov 03 23:14:47.603694 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.603667 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-openshift-ca" pod "44cc9eda-54ea-49fc-80c7-b3138b7abb7b" (UID: "44cc9eda-54ea-49fc-80c7-b3138b7abb7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:14:47.605401 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.605370 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-kube-api-access-z4bm9" pod "44cc9eda-54ea-49fc-80c7-b3138b7abb7b" (UID: "44cc9eda-54ea-49fc-80c7-b3138b7abb7b"). InnerVolumeSpecName "kube-api-access-z4bm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:14:47.605401 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.605389 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-xks-ca" pod "44cc9eda-54ea-49fc-80c7-b3138b7abb7b" (UID: "44cc9eda-54ea-49fc-80c7-b3138b7abb7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:14:47.704091 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.704058 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:14:47.704091 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.704087 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:14:47.704091 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:47.704097 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4bm9\" (UniqueName: \"kubernetes.io/projected/44cc9eda-54ea-49fc-80c7-b3138b7abb7b-kube-api-access-z4bm9\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:14:48.289166 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:48.289126 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" event={"ID":"44cc9eda-54ea-49fc-80c7-b3138b7abb7b","Type":"ContainerDied","Data":"93b49b3ed685600d5f0749111edeeda2d156644228b8830cb62ec2da9118ecff"} Nov 03 23:14:48.289361 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:48.289181 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93b49b3ed685600d5f0749111edeeda2d156644228b8830cb62ec2da9118ecff" Nov 03 23:14:48.289361 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:48.289141 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd" Nov 03 23:14:48.382999 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:14:48.382962 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:14:48.620277 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:48.620194 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn"] Nov 03 23:14:54.348458 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:54.348425 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8"] Nov 03 23:14:56.063556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:56.063522 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd"] Nov 03 23:14:56.074468 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:56.074433 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d5590b-mvqtd"] Nov 03 23:14:57.735643 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:57.735605 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44cc9eda-54ea-49fc-80c7-b3138b7abb7b" path="/var/lib/kubelet/pods/44cc9eda-54ea-49fc-80c7-b3138b7abb7b/volumes" Nov 03 23:14:58.086237 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:58.086143 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:14:59.349442 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:14:59.349399 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" podUID="1bb8be41-533c-4d1d-af6b-c2d636f8db66" containerName="payload-pre-processing" containerID="cri-o://1ae5c706899471c8ca4aef4e77c02ea809aed2f38c7bb8352e47133d93c901c5" gracePeriod=25 Nov 03 23:15:02.934721 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:02.934683 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4"] Nov 03 23:15:02.935250 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:02.934911 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" containerID="cri-o://b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25" gracePeriod=30 Nov 03 23:15:03.641662 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.641622 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" podUID="40bae792-cb9d-49de-8666-b23ae8db65f5" containerName="main" containerID="cri-o://6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72" gracePeriod=45 Nov 03 23:15:03.892008 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.891927 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.977673 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-model-cache\") pod \"40bae792-cb9d-49de-8666-b23ae8db65f5\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.978570 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bae792-cb9d-49de-8666-b23ae8db65f5-kube-api-access-5xsqp\" (UniqueName: \"kubernetes.io/projected/40bae792-cb9d-49de-8666-b23ae8db65f5-kube-api-access-5xsqp\") pod \"40bae792-cb9d-49de-8666-b23ae8db65f5\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.978628 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-home\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-home\") pod \"40bae792-cb9d-49de-8666-b23ae8db65f5\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.978658 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-tmp-dir\") pod \"40bae792-cb9d-49de-8666-b23ae8db65f5\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.978720 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40bae792-cb9d-49de-8666-b23ae8db65f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/40bae792-cb9d-49de-8666-b23ae8db65f5-tls-certs\") pod \"40bae792-cb9d-49de-8666-b23ae8db65f5\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.978740 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-model-cache" pod "40bae792-cb9d-49de-8666-b23ae8db65f5" (UID: "40bae792-cb9d-49de-8666-b23ae8db65f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.978847 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-dshm\") pod \"40bae792-cb9d-49de-8666-b23ae8db65f5\" (UID: \"40bae792-cb9d-49de-8666-b23ae8db65f5\") " Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.978942 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-home" pod "40bae792-cb9d-49de-8666-b23ae8db65f5" (UID: "40bae792-cb9d-49de-8666-b23ae8db65f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.979161 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.979174 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:03.980341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.979460 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-tmp-dir" pod "40bae792-cb9d-49de-8666-b23ae8db65f5" (UID: "40bae792-cb9d-49de-8666-b23ae8db65f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:03.981730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.981692 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-dshm" pod "40bae792-cb9d-49de-8666-b23ae8db65f5" (UID: "40bae792-cb9d-49de-8666-b23ae8db65f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:03.981923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.981901 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bae792-cb9d-49de-8666-b23ae8db65f5-kube-api-access-5xsqp" pod "40bae792-cb9d-49de-8666-b23ae8db65f5" (UID: "40bae792-cb9d-49de-8666-b23ae8db65f5"). InnerVolumeSpecName "kube-api-access-5xsqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:03.982579 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:03.982555 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40bae792-cb9d-49de-8666-b23ae8db65f5-tls-certs" pod "40bae792-cb9d-49de-8666-b23ae8db65f5" (UID: "40bae792-cb9d-49de-8666-b23ae8db65f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:15:04.079885 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.079843 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xsqp\" (UniqueName: \"kubernetes.io/projected/40bae792-cb9d-49de-8666-b23ae8db65f5-kube-api-access-5xsqp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:04.079885 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.079887 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:04.080120 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.079902 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bae792-cb9d-49de-8666-b23ae8db65f5-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:04.080120 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.079913 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bae792-cb9d-49de-8666-b23ae8db65f5-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:04.249500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.249470 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv"] Nov 03 23:15:04.249873 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.249860 4903 cpu_manager.go:414] "containerMap: removing container" podUID="44cc9eda-54ea-49fc-80c7-b3138b7abb7b" containerName="revoke-keys" Nov 03 23:15:04.249920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.249875 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44cc9eda-54ea-49fc-80c7-b3138b7abb7b" containerName="revoke-keys" Nov 03 23:15:04.249920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.249906 4903 cpu_manager.go:414] "containerMap: removing container" podUID="40bae792-cb9d-49de-8666-b23ae8db65f5" containerName="main" Nov 03 23:15:04.249920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.249911 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bae792-cb9d-49de-8666-b23ae8db65f5" containerName="main" Nov 03 23:15:04.250022 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.249988 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bae792-cb9d-49de-8666-b23ae8db65f5" containerName="main" Nov 03 23:15:04.250022 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.250001 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="44cc9eda-54ea-49fc-80c7-b3138b7abb7b" containerName="revoke-keys" Nov 03 23:15:04.253517 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.253444 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.268537 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.268507 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:15:04.268701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.268683 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:15:04.271419 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.271392 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv"] Nov 03 23:15:04.278577 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.278551 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:15:04.278828 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.278808 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:15:04.367240 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.367199 4903 generic.go:356] "Generic (PLEG): container finished" podID="40bae792-cb9d-49de-8666-b23ae8db65f5" containerID="6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72" exitCode=0 Nov 03 23:15:04.367486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.367250 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" Nov 03 23:15:04.367486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.367304 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" event={"ID":"40bae792-cb9d-49de-8666-b23ae8db65f5","Type":"ContainerDied","Data":"6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72"} Nov 03 23:15:04.367486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.367352 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn" event={"ID":"40bae792-cb9d-49de-8666-b23ae8db65f5","Type":"ContainerDied","Data":"ee430e37d425714be289a76f19997c8ac06f821c07f211b2c6ee39c5a8868291"} Nov 03 23:15:04.367486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.367372 4903 scope.go:122] "RemoveContainer" containerID="6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72" Nov 03 23:15:04.377031 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.377010 4903 scope.go:122] "RemoveContainer" containerID="6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72" Nov 03 23:15:04.377350 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:15:04.377330 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72\": container with ID starting with 6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72 not found: ID does not exist" containerID="6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72" Nov 03 23:15:04.377410 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.377359 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72"} err="failed to get container status \"6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72\": rpc error: code = NotFound desc = could not find container \"6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72\": container with ID starting with 6d835888e8c03a84182e9ef5e9e8a0a5119bbdb6dcf24f6a33d03e12fb45ae72 not found: ID does not exist" Nov 03 23:15:04.382372 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.382352 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8bgv\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.382433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.382383 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8bgv\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.382433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.382410 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgjqr\" (UniqueName: \"kubernetes.io/projected/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-kube-api-access-pgjqr\") pod \"maas-api-revoke-sub-3d4156f1-n8bgv\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.444499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.444465 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn"] Nov 03 23:15:04.474484 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.474451 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d5590b/simulator-single-main-d5590b-kserve-7bfd679fb5-c4wxn"] Nov 03 23:15:04.483617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.483584 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8bgv\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.483768 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.483621 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8bgv\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.483768 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.483648 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgjqr\" (UniqueName: \"kubernetes.io/projected/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-kube-api-access-pgjqr\") pod \"maas-api-revoke-sub-3d4156f1-n8bgv\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.484257 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.484238 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8bgv\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.486020 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.486000 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n8bgv\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.519439 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.519355 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgjqr\" (UniqueName: \"kubernetes.io/projected/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-kube-api-access-pgjqr\") pod \"maas-api-revoke-sub-3d4156f1-n8bgv\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.566456 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.566418 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:04.863988 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:04.863964 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv"] Nov 03 23:15:04.866945 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:15:04.866918 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2feee52_65e8_46bc_9d40_3e3e0ee23e68.slice/crio-1f56fcf50f655b1e29eacf92b7459e023ff90e0ce31b435b6ffa5df5f7434309 WatchSource:0}: Error finding container 1f56fcf50f655b1e29eacf92b7459e023ff90e0ce31b435b6ffa5df5f7434309: Status 404 returned error can't find the container with id 1f56fcf50f655b1e29eacf92b7459e023ff90e0ce31b435b6ffa5df5f7434309 Nov 03 23:15:05.374180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:05.374137 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" event={"ID":"a2feee52-65e8-46bc-9d40-3e3e0ee23e68","Type":"ContainerStarted","Data":"7af06d5e5314ed249af1ab4ff787d645f3fa54b1bb64cf68604fd0b3cf5f4a65"} Nov 03 23:15:05.374685 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:05.374197 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" event={"ID":"a2feee52-65e8-46bc-9d40-3e3e0ee23e68","Type":"ContainerStarted","Data":"1f56fcf50f655b1e29eacf92b7459e023ff90e0ce31b435b6ffa5df5f7434309"} Nov 03 23:15:05.410822 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:05.410765 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" podStartSLOduration=1.410750817 podStartE2EDuration="1.410750817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:15:05.409759026 +0000 UTC m=+3731.813700002" watchObservedRunningTime="2026-11-03 23:15:05.410750817 +0000 UTC m=+3731.814691793" Nov 03 23:15:05.714204 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:05.714165 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 23:15:05.714409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:05.714225 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 23:15:05.731246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:05.731207 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bae792-cb9d-49de-8666-b23ae8db65f5" path="/var/lib/kubelet/pods/40bae792-cb9d-49de-8666-b23ae8db65f5/volumes" Nov 03 23:15:06.382308 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:06.382261 4903 generic.go:356] "Generic (PLEG): container finished" podID="a2feee52-65e8-46bc-9d40-3e3e0ee23e68" containerID="7af06d5e5314ed249af1ab4ff787d645f3fa54b1bb64cf68604fd0b3cf5f4a65" exitCode=0 Nov 03 23:15:06.382719 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:06.382332 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" event={"ID":"a2feee52-65e8-46bc-9d40-3e3e0ee23e68","Type":"ContainerDied","Data":"7af06d5e5314ed249af1ab4ff787d645f3fa54b1bb64cf68604fd0b3cf5f4a65"} Nov 03 23:15:07.520957 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.520932 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:07.612202 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.612163 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-kube-api-access-pgjqr\" (UniqueName: \"kubernetes.io/projected/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-kube-api-access-pgjqr\") pod \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " Nov 03 23:15:07.612425 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.612225 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-xks-ca\") pod \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " Nov 03 23:15:07.612425 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.612284 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-openshift-ca\") pod \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\" (UID: \"a2feee52-65e8-46bc-9d40-3e3e0ee23e68\") " Nov 03 23:15:07.612608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.612585 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-openshift-ca" pod "a2feee52-65e8-46bc-9d40-3e3e0ee23e68" (UID: "a2feee52-65e8-46bc-9d40-3e3e0ee23e68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:07.612693 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.612672 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:07.614431 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.614410 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-kube-api-access-pgjqr" pod "a2feee52-65e8-46bc-9d40-3e3e0ee23e68" (UID: "a2feee52-65e8-46bc-9d40-3e3e0ee23e68"). InnerVolumeSpecName "kube-api-access-pgjqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:07.614514 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.614462 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-xks-ca" pod "a2feee52-65e8-46bc-9d40-3e3e0ee23e68" (UID: "a2feee52-65e8-46bc-9d40-3e3e0ee23e68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:15:07.714210 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.714172 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgjqr\" (UniqueName: \"kubernetes.io/projected/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-kube-api-access-pgjqr\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:07.714210 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.714203 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2feee52-65e8-46bc-9d40-3e3e0ee23e68-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:07.949516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:07.949483 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" podUID="1bb8be41-533c-4d1d-af6b-c2d636f8db66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:15:08.180846 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.180818 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:15:08.318882 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.318802 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ecf6f8-6379-41c5-9d51-b30162893bc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ecf6f8-6379-41c5-9d51-b30162893bc8-istiod-ca-cert\") pod \"43ecf6f8-6379-41c5-9d51-b30162893bc8\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " Nov 03 23:15:08.318882 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.318845 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-kube-api-access-xsb82\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-kube-api-access-xsb82\") pod \"43ecf6f8-6379-41c5-9d51-b30162893bc8\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " Nov 03 23:15:08.319107 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.318882 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-data\") pod \"43ecf6f8-6379-41c5-9d51-b30162893bc8\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " Nov 03 23:15:08.319107 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.318908 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-certs\") pod \"43ecf6f8-6379-41c5-9d51-b30162893bc8\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " Nov 03 23:15:08.319107 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.318945 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-envoy\") pod \"43ecf6f8-6379-41c5-9d51-b30162893bc8\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " Nov 03 23:15:08.319107 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319024 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-socket\") pod \"43ecf6f8-6379-41c5-9d51-b30162893bc8\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " Nov 03 23:15:08.319107 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319058 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-podinfo\") pod \"43ecf6f8-6379-41c5-9d51-b30162893bc8\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " Nov 03 23:15:08.319107 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319089 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-token\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-token\") pod \"43ecf6f8-6379-41c5-9d51-b30162893bc8\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " Nov 03 23:15:08.319452 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319135 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-credential-socket\") pod \"43ecf6f8-6379-41c5-9d51-b30162893bc8\" (UID: \"43ecf6f8-6379-41c5-9d51-b30162893bc8\") " Nov 03 23:15:08.319452 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319186 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-data" pod "43ecf6f8-6379-41c5-9d51-b30162893bc8" (UID: "43ecf6f8-6379-41c5-9d51-b30162893bc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:08.319452 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319233 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ecf6f8-6379-41c5-9d51-b30162893bc8-istiod-ca-cert" pod "43ecf6f8-6379-41c5-9d51-b30162893bc8" (UID: "43ecf6f8-6379-41c5-9d51-b30162893bc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:08.319452 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319433 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-socket" pod "43ecf6f8-6379-41c5-9d51-b30162893bc8" (UID: "43ecf6f8-6379-41c5-9d51-b30162893bc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:08.319595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319516 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-certs" pod "43ecf6f8-6379-41c5-9d51-b30162893bc8" (UID: "43ecf6f8-6379-41c5-9d51-b30162893bc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:08.319909 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319874 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ecf6f8-6379-41c5-9d51-b30162893bc8-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:08.321991 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.319915 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:08.321991 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.320244 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-credential-socket" pod "43ecf6f8-6379-41c5-9d51-b30162893bc8" (UID: "43ecf6f8-6379-41c5-9d51-b30162893bc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:08.328620 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.322751 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-podinfo" pod "43ecf6f8-6379-41c5-9d51-b30162893bc8" (UID: "43ecf6f8-6379-41c5-9d51-b30162893bc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:15:08.329417 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.329391 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-kube-api-access-xsb82" pod "43ecf6f8-6379-41c5-9d51-b30162893bc8" (UID: "43ecf6f8-6379-41c5-9d51-b30162893bc8"). InnerVolumeSpecName "kube-api-access-xsb82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:08.329628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.329600 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-envoy" pod "43ecf6f8-6379-41c5-9d51-b30162893bc8" (UID: "43ecf6f8-6379-41c5-9d51-b30162893bc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:15:08.330095 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.330081 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-token" pod "43ecf6f8-6379-41c5-9d51-b30162893bc8" (UID: "43ecf6f8-6379-41c5-9d51-b30162893bc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:08.393720 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.393681 4903 generic.go:356] "Generic (PLEG): container finished" podID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerID="b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25" exitCode=0 Nov 03 23:15:08.393908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.393753 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" Nov 03 23:15:08.393908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.393770 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" event={"ID":"43ecf6f8-6379-41c5-9d51-b30162893bc8","Type":"ContainerDied","Data":"b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25"} Nov 03 23:15:08.393908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.393851 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4" event={"ID":"43ecf6f8-6379-41c5-9d51-b30162893bc8","Type":"ContainerDied","Data":"fce15310b270fdeea654dde09bd6babd4eff1377c8915bc2a9f259207f96b7fe"} Nov 03 23:15:08.393908 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.393884 4903 scope.go:122] "RemoveContainer" containerID="b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25" Nov 03 23:15:08.395256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.395226 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" Nov 03 23:15:08.395256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.395231 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv" event={"ID":"a2feee52-65e8-46bc-9d40-3e3e0ee23e68","Type":"ContainerDied","Data":"1f56fcf50f655b1e29eacf92b7459e023ff90e0ce31b435b6ffa5df5f7434309"} Nov 03 23:15:08.395418 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.395277 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f56fcf50f655b1e29eacf92b7459e023ff90e0ce31b435b6ffa5df5f7434309" Nov 03 23:15:08.403334 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.403288 4903 scope.go:122] "RemoveContainer" containerID="22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9" Nov 03 23:15:08.411273 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.411255 4903 scope.go:122] "RemoveContainer" containerID="b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25" Nov 03 23:15:08.411537 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:15:08.411518 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25\": container with ID starting with b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25 not found: ID does not exist" containerID="b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25" Nov 03 23:15:08.411614 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.411544 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25"} err="failed to get container status \"b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25\": rpc error: code = NotFound desc = could not find container \"b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25\": container with ID starting with b47408f6931dedb0faebdbd8a89acef0342fa60303c8f5655aa564361e74ae25 not found: ID does not exist" Nov 03 23:15:08.411614 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.411562 4903 scope.go:122] "RemoveContainer" containerID="22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9" Nov 03 23:15:08.411791 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:15:08.411772 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9\": container with ID starting with 22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9 not found: ID does not exist" containerID="22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9" Nov 03 23:15:08.411835 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.411798 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9"} err="failed to get container status \"22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9\": rpc error: code = NotFound desc = could not find container \"22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9\": container with ID starting with 22c21c8a1931da804f023c2ae874f6a7e18aa8a8d1da9590e1ed318a693e6db9 not found: ID does not exist" Nov 03 23:15:08.420799 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.420777 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsb82\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-kube-api-access-xsb82\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:08.420890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.420805 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:08.420890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.420816 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:08.420890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.420823 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:08.420890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.420831 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:08.420890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.420839 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ecf6f8-6379-41c5-9d51-b30162893bc8-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:08.420890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.420847 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ecf6f8-6379-41c5-9d51-b30162893bc8-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:08.454920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.454881 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4"] Nov 03 23:15:08.460932 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:08.460900 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d5590b-openshift-default-7bd898b6c7rlgk4"] Nov 03 23:15:09.732395 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:09.732354 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" path="/var/lib/kubelet/pods/43ecf6f8-6379-41c5-9d51-b30162893bc8/volumes" Nov 03 23:15:16.541280 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:16.541248 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:15:17.966318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:17.965378 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" podUID="1bb8be41-533c-4d1d-af6b-c2d636f8db66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 23:15:18.437462 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:15:18.437411 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:15:19.525866 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:19.525652 4903 generic.go:356] "Generic (PLEG): container finished" podID="1bb8be41-533c-4d1d-af6b-c2d636f8db66" containerID="1ae5c706899471c8ca4aef4e77c02ea809aed2f38c7bb8352e47133d93c901c5" exitCode=0 Nov 03 23:15:19.525866 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:19.525753 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" event={"ID":"1bb8be41-533c-4d1d-af6b-c2d636f8db66","Type":"ContainerDied","Data":"1ae5c706899471c8ca4aef4e77c02ea809aed2f38c7bb8352e47133d93c901c5"} Nov 03 23:15:19.597540 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:19.597512 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:15:19.735820 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:19.735769 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bb8be41-533c-4d1d-af6b-c2d636f8db66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb8be41-533c-4d1d-af6b-c2d636f8db66-plugins-config-volume\") pod \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\" (UID: \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\") " Nov 03 23:15:19.736021 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:19.735828 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bb8be41-533c-4d1d-af6b-c2d636f8db66-kube-api-access-jtphg\" (UniqueName: \"kubernetes.io/projected/1bb8be41-533c-4d1d-af6b-c2d636f8db66-kube-api-access-jtphg\") pod \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\" (UID: \"1bb8be41-533c-4d1d-af6b-c2d636f8db66\") " Nov 03 23:15:19.736228 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:19.736198 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb8be41-533c-4d1d-af6b-c2d636f8db66-plugins-config-volume" pod "1bb8be41-533c-4d1d-af6b-c2d636f8db66" (UID: "1bb8be41-533c-4d1d-af6b-c2d636f8db66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:15:19.740483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:19.740439 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bb8be41-533c-4d1d-af6b-c2d636f8db66-kube-api-access-jtphg" pod "1bb8be41-533c-4d1d-af6b-c2d636f8db66" (UID: "1bb8be41-533c-4d1d-af6b-c2d636f8db66"). InnerVolumeSpecName "kube-api-access-jtphg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:15:19.838140 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:19.838100 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bb8be41-533c-4d1d-af6b-c2d636f8db66-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:19.838140 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:19.838134 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtphg\" (UniqueName: \"kubernetes.io/projected/1bb8be41-533c-4d1d-af6b-c2d636f8db66-kube-api-access-jtphg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:15:20.532402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:20.532372 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" Nov 03 23:15:20.532402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:20.532385 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8" event={"ID":"1bb8be41-533c-4d1d-af6b-c2d636f8db66","Type":"ContainerDied","Data":"6f0dba651bb1cc07fb7ad973becc163c022173f30fe4662011c4bab2073266d5"} Nov 03 23:15:20.532974 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:20.532454 4903 scope.go:122] "RemoveContainer" containerID="1ae5c706899471c8ca4aef4e77c02ea809aed2f38c7bb8352e47133d93c901c5" Nov 03 23:15:20.577351 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:20.577321 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8"] Nov 03 23:15:20.588521 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:20.588493 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d5590b-5575b4pmf8"] Nov 03 23:15:21.732489 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:21.732455 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bb8be41-533c-4d1d-af6b-c2d636f8db66" path="/var/lib/kubelet/pods/1bb8be41-533c-4d1d-af6b-c2d636f8db66/volumes" Nov 03 23:15:21.815595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:15:21.815564 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:15:48.383166 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:15:48.383128 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:16:18.382864 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:16:18.382826 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:16:48.382969 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:16:48.382892 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:16:49.190849 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:16:49.190807 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:17:04.711440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:04.711372 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:17:09.948874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:09.948814 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:17:18.382817 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:17:18.382774 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:17:25.781474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:25.781439 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:17:26.571181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.571146 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5"] Nov 03 23:17:26.571805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.571777 4903 cpu_manager.go:414] "containerMap: removing container" podUID="a2feee52-65e8-46bc-9d40-3e3e0ee23e68" containerName="revoke-keys" Nov 03 23:17:26.571805 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.571800 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2feee52-65e8-46bc-9d40-3e3e0ee23e68" containerName="revoke-keys" Nov 03 23:17:26.572029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.571830 4903 cpu_manager.go:414] "containerMap: removing container" podUID="1bb8be41-533c-4d1d-af6b-c2d636f8db66" containerName="payload-pre-processing" Nov 03 23:17:26.572029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.571838 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bb8be41-533c-4d1d-af6b-c2d636f8db66" containerName="payload-pre-processing" Nov 03 23:17:26.572029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.571856 4903 cpu_manager.go:414] "containerMap: removing container" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" Nov 03 23:17:26.572029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.571863 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" Nov 03 23:17:26.572029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.571976 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bb8be41-533c-4d1d-af6b-c2d636f8db66" containerName="payload-pre-processing" Nov 03 23:17:26.572029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.571989 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" Nov 03 23:17:26.572029 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.572004 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2feee52-65e8-46bc-9d40-3e3e0ee23e68" containerName="revoke-keys" Nov 03 23:17:26.575770 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.575747 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.600788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.600756 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5"] Nov 03 23:17:26.604740 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.604715 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-22c413-openshift-default-dockercfg-dwqk6\"" Nov 03 23:17:26.675935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.675896 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-data\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.676112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.675941 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26m7t\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-kube-api-access-26m7t\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.676112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.676019 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-certs\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.676112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.676088 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-envoy\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.676231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.676123 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-credential-socket\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.676231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.676143 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-socket\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.676231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.676161 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9279d7e7-d221-4df4-878b-2e5d92ae6490-istiod-ca-cert\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.676354 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.676270 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-token\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.676354 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.676329 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-podinfo\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.776941 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.776908 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-data\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.776969 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26m7t\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-kube-api-access-26m7t\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777026 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-certs\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777048 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-envoy\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777086 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-credential-socket\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777117 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-socket\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777143 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9279d7e7-d221-4df4-878b-2e5d92ae6490-istiod-ca-cert\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777206 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777190 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-token\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777345 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-podinfo\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777451 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-data\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777501 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-certs\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777574 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-socket\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.777897 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.777722 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-credential-socket\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.778098 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.778077 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9279d7e7-d221-4df4-878b-2e5d92ae6490-istiod-ca-cert\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.779529 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.779506 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-envoy\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.779863 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.779841 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-podinfo\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.805088 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.805061 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-token\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.805879 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.805861 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26m7t\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-kube-api-access-26m7t\") pod \"e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:26.890046 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:26.889977 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:27.213878 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:27.213848 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5"] Nov 03 23:17:27.216772 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:17:27.216742 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9279d7e7_d221_4df4_878b_2e5d92ae6490.slice/crio-1c6c7fd96cc7b0c75f04749388967c91734ebd777bf237947f884324f2c4c7aa WatchSource:0}: Error finding container 1c6c7fd96cc7b0c75f04749388967c91734ebd777bf237947f884324f2c4c7aa: Status 404 returned error can't find the container with id 1c6c7fd96cc7b0c75f04749388967c91734ebd777bf237947f884324f2c4c7aa Nov 03 23:17:27.218850 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:27.218812 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:17:27.218914 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:27.218882 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:17:27.218914 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:27.218910 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:17:28.178921 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:28.178884 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" event={"ID":"9279d7e7-d221-4df4-878b-2e5d92ae6490","Type":"ContainerStarted","Data":"c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e"} Nov 03 23:17:28.179414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:28.178935 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" event={"ID":"9279d7e7-d221-4df4-878b-2e5d92ae6490","Type":"ContainerStarted","Data":"1c6c7fd96cc7b0c75f04749388967c91734ebd777bf237947f884324f2c4c7aa"} Nov 03 23:17:28.231037 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:28.230986 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" podStartSLOduration=2.230972099 podStartE2EDuration="2.230972099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:28.230454323 +0000 UTC m=+3874.634395311" watchObservedRunningTime="2026-11-03 23:17:28.230972099 +0000 UTC m=+3874.634913074" Nov 03 23:17:28.890127 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:28.890091 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:28.895318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:28.895276 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:29.183843 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:29.183816 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:29.184983 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:29.184963 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:32.355492 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.355459 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl"] Nov 03 23:17:32.355903 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.355878 4903 cpu_manager.go:414] "containerMap: removing container" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" Nov 03 23:17:32.355950 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.355904 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" Nov 03 23:17:32.356003 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.355992 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ecf6f8-6379-41c5-9d51-b30162893bc8" containerName="istio-proxy" Nov 03 23:17:32.359059 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.359042 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:32.376510 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.376483 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl"] Nov 03 23:17:32.384734 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.384713 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-22c413\"" Nov 03 23:17:32.429121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.429095 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4t27\" (UniqueName: \"kubernetes.io/projected/b4a9af3f-b527-443c-8caa-8104a4052805-kube-api-access-l4t27\") pod \"payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl\" (UID: \"b4a9af3f-b527-443c-8caa-8104a4052805\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:32.429272 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.429218 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a9af3f-b527-443c-8caa-8104a4052805-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl\" (UID: \"b4a9af3f-b527-443c-8caa-8104a4052805\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:32.530142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.530104 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4t27\" (UniqueName: \"kubernetes.io/projected/b4a9af3f-b527-443c-8caa-8104a4052805-kube-api-access-l4t27\") pod \"payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl\" (UID: \"b4a9af3f-b527-443c-8caa-8104a4052805\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:32.530361 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.530241 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a9af3f-b527-443c-8caa-8104a4052805-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl\" (UID: \"b4a9af3f-b527-443c-8caa-8104a4052805\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:32.531087 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.531061 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a9af3f-b527-443c-8caa-8104a4052805-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl\" (UID: \"b4a9af3f-b527-443c-8caa-8104a4052805\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:32.551491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.551458 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4t27\" (UniqueName: \"kubernetes.io/projected/b4a9af3f-b527-443c-8caa-8104a4052805-kube-api-access-l4t27\") pod \"payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl\" (UID: \"b4a9af3f-b527-443c-8caa-8104a4052805\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:32.672152 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.672119 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:32.981686 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:32.981578 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl"] Nov 03 23:17:32.984033 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:17:32.984002 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a9af3f_b527_443c_8caa_8104a4052805.slice/crio-e633c1c339cd065a22e9d262ddedc1c2f05d46fab8db903896fca35f5af0d62a WatchSource:0}: Error finding container e633c1c339cd065a22e9d262ddedc1c2f05d46fab8db903896fca35f5af0d62a: Status 404 returned error can't find the container with id e633c1c339cd065a22e9d262ddedc1c2f05d46fab8db903896fca35f5af0d62a Nov 03 23:17:33.204191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:33.204154 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" event={"ID":"b4a9af3f-b527-443c-8caa-8104a4052805","Type":"ContainerStarted","Data":"3eb0780ccf9dde10d400ffc5f0afb20e4ef45080e4df6b382821d95d959e2688"} Nov 03 23:17:33.204408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:33.204217 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" event={"ID":"b4a9af3f-b527-443c-8caa-8104a4052805","Type":"ContainerStarted","Data":"e633c1c339cd065a22e9d262ddedc1c2f05d46fab8db903896fca35f5af0d62a"} Nov 03 23:17:33.204408 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:33.204315 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:33.241939 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:33.241828 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" podStartSLOduration=1.241812161 podStartE2EDuration="1.241812161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:33.240266061 +0000 UTC m=+3879.644207033" watchObservedRunningTime="2026-11-03 23:17:33.241812161 +0000 UTC m=+3879.645753137" Nov 03 23:17:36.304342 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:36.304309 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:17:44.212235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:44.212202 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:17:45.225463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:45.225424 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:17:45.225840 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:45.225495 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:17:46.290045 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:46.290009 4903 generic.go:356] "Generic (PLEG): container finished" podID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerID="c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e" exitCode=0 Nov 03 23:17:46.290541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:46.290063 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" event={"ID":"9279d7e7-d221-4df4-878b-2e5d92ae6490","Type":"ContainerDied","Data":"c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e"} Nov 03 23:17:46.290541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:46.290476 4903 scope.go:122] "RemoveContainer" containerID="c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e" Nov 03 23:17:46.291711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:46.291683 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:17:46.291776 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:46.291755 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:17:46.291827 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:46.291784 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:17:47.296343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.296287 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" event={"ID":"9279d7e7-d221-4df4-878b-2e5d92ae6490","Type":"ContainerStarted","Data":"af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890"} Nov 03 23:17:47.486780 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.486744 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl"] Nov 03 23:17:47.489377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.489358 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.502513 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.502487 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-22c413\"/\"simulator-single-main-22c413-kserve-self-signed-certs\"" Nov 03 23:17:47.502640 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.502512 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-22c413\"/\"openshift-service-ca.crt\"" Nov 03 23:17:47.502640 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.502516 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-22c413\"/\"kube-root-ca.crt\"" Nov 03 23:17:47.502640 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.502576 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-22c413\"/\"default-dockercfg-rltmr\"" Nov 03 23:17:47.508050 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.508021 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl"] Nov 03 23:17:47.585104 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.585017 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-home\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.585104 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.585061 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f286c9-c002-46a0-b24e-86a51cd21ddd-tls-certs\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.585329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.585172 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-dshm\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.585329 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.585205 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxvk\" (UniqueName: \"kubernetes.io/projected/04f286c9-c002-46a0-b24e-86a51cd21ddd-kube-api-access-8fxvk\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.585414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.585365 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-model-cache\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.585414 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.585403 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-tmp-dir\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.686090 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.686053 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-dshm\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.686090 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.686091 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxvk\" (UniqueName: \"kubernetes.io/projected/04f286c9-c002-46a0-b24e-86a51cd21ddd-kube-api-access-8fxvk\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.686388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.686153 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-model-cache\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.686388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.686181 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-tmp-dir\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.686388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.686208 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-home\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.686388 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.686303 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f286c9-c002-46a0-b24e-86a51cd21ddd-tls-certs\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.686619 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.686585 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-model-cache\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.686688 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.686632 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-home\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.686756 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.686725 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-tmp-dir\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.688453 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.688433 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-dshm\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.688844 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.688820 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f286c9-c002-46a0-b24e-86a51cd21ddd-tls-certs\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.709742 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.709716 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxvk\" (UniqueName: \"kubernetes.io/projected/04f286c9-c002-46a0-b24e-86a51cd21ddd-kube-api-access-8fxvk\") pod \"simulator-single-main-22c413-kserve-84486f75d5-4z9xl\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.803216 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.803173 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:47.891478 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.891070 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:47.891478 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.891180 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" start-of-body= Nov 03 23:17:47.891478 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:47.891213 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" Nov 03 23:17:48.123571 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:48.123538 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl"] Nov 03 23:17:48.126733 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:17:48.126664 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04f286c9_c002_46a0_b24e_86a51cd21ddd.slice/crio-8fcd8d8538ecb14809f6013ec5a619e44e686cafd1e292e44f2ba6b40c1665d6 WatchSource:0}: Error finding container 8fcd8d8538ecb14809f6013ec5a619e44e686cafd1e292e44f2ba6b40c1665d6: Status 404 returned error can't find the container with id 8fcd8d8538ecb14809f6013ec5a619e44e686cafd1e292e44f2ba6b40c1665d6 Nov 03 23:17:48.302697 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:48.302651 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" event={"ID":"04f286c9-c002-46a0-b24e-86a51cd21ddd","Type":"ContainerStarted","Data":"219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c"} Nov 03 23:17:48.302697 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:48.302704 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" event={"ID":"04f286c9-c002-46a0-b24e-86a51cd21ddd","Type":"ContainerStarted","Data":"8fcd8d8538ecb14809f6013ec5a619e44e686cafd1e292e44f2ba6b40c1665d6"} Nov 03 23:17:48.351318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:48.346768 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" podStartSLOduration=1.346749417 podStartE2EDuration="1.346749417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:17:48.346519733 +0000 UTC m=+3894.750460709" watchObservedRunningTime="2026-11-03 23:17:48.346749417 +0000 UTC m=+3894.750690390" Nov 03 23:17:48.390438 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:17:48.390355 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:17:48.803549 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:48.803501 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:48.891112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:48.891074 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" start-of-body= Nov 03 23:17:48.891315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:48.891139 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" Nov 03 23:17:49.102422 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:49.102347 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:17:49.890381 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:49.890346 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" start-of-body= Nov 03 23:17:49.890901 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:49.890874 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" Nov 03 23:17:50.890762 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:50.890725 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" start-of-body= Nov 03 23:17:50.891170 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:50.890780 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" Nov 03 23:17:51.890472 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:51.890434 4903 patch_prober.go:29] interesting pod/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" start-of-body= Nov 03 23:17:51.890668 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:51.890498 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.147:15021/healthz/ready\": dial tcp 10.132.1.147:15021: connect: connection refused" Nov 03 23:17:52.894010 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:52.893976 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:52.894435 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:52.894321 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:52.894935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:52.894917 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:17:54.493873 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:54.493832 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:17:54.501112 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:54.501089 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:17:54.502744 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:54.502724 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:17:54.510030 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:54.510014 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:17:57.803677 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:57.803638 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:57.816036 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:57.816008 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:17:58.364865 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:17:58.364835 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:18:05.898268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:05.898230 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:18:17.844587 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.844552 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq"] Nov 03 23:18:17.848427 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.848407 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:17.873989 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.873961 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:18:17.874218 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.874195 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:18:17.874370 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.874218 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:18:17.874370 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.874224 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:18:17.876024 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.876006 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq"] Nov 03 23:18:17.982064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.982031 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b617c1d-s5bfq\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:17.982242 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.982074 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b617c1d-s5bfq\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:17.982242 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:17.982205 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fx7g\" (UniqueName: \"kubernetes.io/projected/cc640593-e132-47fa-adfb-f3d02ad55962-kube-api-access-4fx7g\") pod \"maas-api-revoke-sub-6b617c1d-s5bfq\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:18.083191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:18.083151 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b617c1d-s5bfq\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:18.083409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:18.083212 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b617c1d-s5bfq\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:18.083409 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:18.083353 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fx7g\" (UniqueName: \"kubernetes.io/projected/cc640593-e132-47fa-adfb-f3d02ad55962-kube-api-access-4fx7g\") pod \"maas-api-revoke-sub-6b617c1d-s5bfq\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:18.083884 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:18.083855 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b617c1d-s5bfq\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:18.085757 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:18.085730 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b617c1d-s5bfq\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:18.107349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:18.107271 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fx7g\" (UniqueName: \"kubernetes.io/projected/cc640593-e132-47fa-adfb-f3d02ad55962-kube-api-access-4fx7g\") pod \"maas-api-revoke-sub-6b617c1d-s5bfq\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:18.161272 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:18.161240 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:18.383245 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:18:18.383154 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:18:18.456692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:18.456667 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq"] Nov 03 23:18:18.459234 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:18:18.459202 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc640593_e132_47fa_adfb_f3d02ad55962.slice/crio-ef4a87d57cf0c7678806fb712b79d6361a918becd7474ac15a8492f57ae1799c WatchSource:0}: Error finding container ef4a87d57cf0c7678806fb712b79d6361a918becd7474ac15a8492f57ae1799c: Status 404 returned error can't find the container with id ef4a87d57cf0c7678806fb712b79d6361a918becd7474ac15a8492f57ae1799c Nov 03 23:18:19.458231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:19.458196 4903 generic.go:356] "Generic (PLEG): container finished" podID="cc640593-e132-47fa-adfb-f3d02ad55962" containerID="93161df3ddd0b0f6cff88125a2b2623c0e0776c475a465ad344bb79d2951af5b" exitCode=0 Nov 03 23:18:19.458648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:19.458286 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" event={"ID":"cc640593-e132-47fa-adfb-f3d02ad55962","Type":"ContainerDied","Data":"93161df3ddd0b0f6cff88125a2b2623c0e0776c475a465ad344bb79d2951af5b"} Nov 03 23:18:19.458648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:19.458365 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" event={"ID":"cc640593-e132-47fa-adfb-f3d02ad55962","Type":"ContainerStarted","Data":"ef4a87d57cf0c7678806fb712b79d6361a918becd7474ac15a8492f57ae1799c"} Nov 03 23:18:20.601421 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.601396 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:20.711057 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.711020 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-openshift-ca\") pod \"cc640593-e132-47fa-adfb-f3d02ad55962\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " Nov 03 23:18:20.711266 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.711073 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-xks-ca\") pod \"cc640593-e132-47fa-adfb-f3d02ad55962\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " Nov 03 23:18:20.711266 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.711153 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc640593-e132-47fa-adfb-f3d02ad55962-kube-api-access-4fx7g\" (UniqueName: \"kubernetes.io/projected/cc640593-e132-47fa-adfb-f3d02ad55962-kube-api-access-4fx7g\") pod \"cc640593-e132-47fa-adfb-f3d02ad55962\" (UID: \"cc640593-e132-47fa-adfb-f3d02ad55962\") " Nov 03 23:18:20.711489 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.711464 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-openshift-ca" pod "cc640593-e132-47fa-adfb-f3d02ad55962" (UID: "cc640593-e132-47fa-adfb-f3d02ad55962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:20.713383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.713311 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc640593-e132-47fa-adfb-f3d02ad55962-kube-api-access-4fx7g" pod "cc640593-e132-47fa-adfb-f3d02ad55962" (UID: "cc640593-e132-47fa-adfb-f3d02ad55962"). InnerVolumeSpecName "kube-api-access-4fx7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:20.713383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.713337 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-xks-ca" pod "cc640593-e132-47fa-adfb-f3d02ad55962" (UID: "cc640593-e132-47fa-adfb-f3d02ad55962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:18:20.812730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.812695 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fx7g\" (UniqueName: \"kubernetes.io/projected/cc640593-e132-47fa-adfb-f3d02ad55962-kube-api-access-4fx7g\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:20.812730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.812724 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:20.812730 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:20.812735 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc640593-e132-47fa-adfb-f3d02ad55962-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:21.468464 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:21.468428 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" Nov 03 23:18:21.468464 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:21.468456 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq" event={"ID":"cc640593-e132-47fa-adfb-f3d02ad55962","Type":"ContainerDied","Data":"ef4a87d57cf0c7678806fb712b79d6361a918becd7474ac15a8492f57ae1799c"} Nov 03 23:18:21.468700 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:21.468504 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef4a87d57cf0c7678806fb712b79d6361a918becd7474ac15a8492f57ae1799c" Nov 03 23:18:28.904208 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.904171 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54"] Nov 03 23:18:28.904681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.904631 4903 cpu_manager.go:414] "containerMap: removing container" podUID="cc640593-e132-47fa-adfb-f3d02ad55962" containerName="revoke-keys" Nov 03 23:18:28.904681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.904644 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc640593-e132-47fa-adfb-f3d02ad55962" containerName="revoke-keys" Nov 03 23:18:28.904772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.904727 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc640593-e132-47fa-adfb-f3d02ad55962" containerName="revoke-keys" Nov 03 23:18:28.907923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.907905 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:28.923130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.923107 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:18:28.933500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.933479 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:18:28.933679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.933479 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:18:28.933679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.933528 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:18:28.935455 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:28.935433 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54"] Nov 03 23:18:29.002491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.002461 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.002652 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.002578 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjsd\" (UniqueName: \"kubernetes.io/projected/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-kube-api-access-qpjsd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.002652 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.002622 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.103464 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.103431 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjsd\" (UniqueName: \"kubernetes.io/projected/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-kube-api-access-qpjsd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.103630 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.103479 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.103630 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.103563 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.104412 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.104382 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.105919 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.105893 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.124246 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.124221 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjsd\" (UniqueName: \"kubernetes.io/projected/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-kube-api-access-qpjsd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.220515 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.220429 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:29.396265 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.396230 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:18:29.516357 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:29.516332 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54"] Nov 03 23:18:29.519137 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:18:29.519111 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc97aa8c_9ce6_469b_8ed2_376a731efb6c.slice/crio-f4ed09878f5e023744ffafea39369bb950dcb86dae497ed97cf463ebf924fca4 WatchSource:0}: Error finding container f4ed09878f5e023744ffafea39369bb950dcb86dae497ed97cf463ebf924fca4: Status 404 returned error can't find the container with id f4ed09878f5e023744ffafea39369bb950dcb86dae497ed97cf463ebf924fca4 Nov 03 23:18:30.512079 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:30.512043 4903 generic.go:356] "Generic (PLEG): container finished" podID="dc97aa8c-9ce6-469b-8ed2-376a731efb6c" containerID="4d8ed773180e613d4ee4fee524b663cf9dfedbcae79f7f0bbefb69ec8390991a" exitCode=0 Nov 03 23:18:30.512534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:30.512134 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" event={"ID":"dc97aa8c-9ce6-469b-8ed2-376a731efb6c","Type":"ContainerDied","Data":"4d8ed773180e613d4ee4fee524b663cf9dfedbcae79f7f0bbefb69ec8390991a"} Nov 03 23:18:30.512534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:30.512190 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" event={"ID":"dc97aa8c-9ce6-469b-8ed2-376a731efb6c","Type":"ContainerStarted","Data":"f4ed09878f5e023744ffafea39369bb950dcb86dae497ed97cf463ebf924fca4"} Nov 03 23:18:31.651158 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.651132 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:31.830458 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.830373 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-xks-ca\") pod \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " Nov 03 23:18:31.830617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.830547 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-kube-api-access-qpjsd\" (UniqueName: \"kubernetes.io/projected/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-kube-api-access-qpjsd\") pod \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " Nov 03 23:18:31.830664 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.830630 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-openshift-ca\") pod \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\" (UID: \"dc97aa8c-9ce6-469b-8ed2-376a731efb6c\") " Nov 03 23:18:31.830958 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.830933 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-openshift-ca" pod "dc97aa8c-9ce6-469b-8ed2-376a731efb6c" (UID: "dc97aa8c-9ce6-469b-8ed2-376a731efb6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:31.832487 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.832465 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-xks-ca" pod "dc97aa8c-9ce6-469b-8ed2-376a731efb6c" (UID: "dc97aa8c-9ce6-469b-8ed2-376a731efb6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:18:31.832749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.832734 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-kube-api-access-qpjsd" pod "dc97aa8c-9ce6-469b-8ed2-376a731efb6c" (UID: "dc97aa8c-9ce6-469b-8ed2-376a731efb6c"). InnerVolumeSpecName "kube-api-access-qpjsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:31.932110 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.932072 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpjsd\" (UniqueName: \"kubernetes.io/projected/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-kube-api-access-qpjsd\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:31.932110 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.932103 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:31.932110 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:31.932113 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc97aa8c-9ce6-469b-8ed2-376a731efb6c-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:32.522450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:32.522420 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" Nov 03 23:18:32.522450 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:32.522428 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54" event={"ID":"dc97aa8c-9ce6-469b-8ed2-376a731efb6c","Type":"ContainerDied","Data":"f4ed09878f5e023744ffafea39369bb950dcb86dae497ed97cf463ebf924fca4"} Nov 03 23:18:32.522675 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:32.522474 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4ed09878f5e023744ffafea39369bb950dcb86dae497ed97cf463ebf924fca4" Nov 03 23:18:33.672768 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:33.672736 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl"] Nov 03 23:18:39.342074 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:39.342038 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl"] Nov 03 23:18:40.708619 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:40.708579 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54"] Nov 03 23:18:40.718490 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:40.718462 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22c413-srh54"] Nov 03 23:18:41.731717 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:41.731680 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc97aa8c-9ce6-469b-8ed2-376a731efb6c" path="/var/lib/kubelet/pods/dc97aa8c-9ce6-469b-8ed2-376a731efb6c/volumes" Nov 03 23:18:42.627357 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:42.627317 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:18:44.342418 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.342360 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" podUID="b4a9af3f-b527-443c-8caa-8104a4052805" containerName="payload-processing" containerID="cri-o://3eb0780ccf9dde10d400ffc5f0afb20e4ef45080e4df6b382821d95d959e2688" gracePeriod=25 Nov 03 23:18:44.578008 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.577980 4903 generic.go:356] "Generic (PLEG): container finished" podID="b4a9af3f-b527-443c-8caa-8104a4052805" containerID="3eb0780ccf9dde10d400ffc5f0afb20e4ef45080e4df6b382821d95d959e2688" exitCode=0 Nov 03 23:18:44.578143 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.578057 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" event={"ID":"b4a9af3f-b527-443c-8caa-8104a4052805","Type":"ContainerDied","Data":"3eb0780ccf9dde10d400ffc5f0afb20e4ef45080e4df6b382821d95d959e2688"} Nov 03 23:18:44.578143 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.578124 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" event={"ID":"b4a9af3f-b527-443c-8caa-8104a4052805","Type":"ContainerDied","Data":"e633c1c339cd065a22e9d262ddedc1c2f05d46fab8db903896fca35f5af0d62a"} Nov 03 23:18:44.578234 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.578148 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e633c1c339cd065a22e9d262ddedc1c2f05d46fab8db903896fca35f5af0d62a" Nov 03 23:18:44.589567 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.589548 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:18:44.651500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.651474 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a9af3f-b527-443c-8caa-8104a4052805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a9af3f-b527-443c-8caa-8104a4052805-plugins-config-volume\") pod \"b4a9af3f-b527-443c-8caa-8104a4052805\" (UID: \"b4a9af3f-b527-443c-8caa-8104a4052805\") " Nov 03 23:18:44.651656 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.651532 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a9af3f-b527-443c-8caa-8104a4052805-kube-api-access-l4t27\" (UniqueName: \"kubernetes.io/projected/b4a9af3f-b527-443c-8caa-8104a4052805-kube-api-access-l4t27\") pod \"b4a9af3f-b527-443c-8caa-8104a4052805\" (UID: \"b4a9af3f-b527-443c-8caa-8104a4052805\") " Nov 03 23:18:44.651927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.651901 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a9af3f-b527-443c-8caa-8104a4052805-plugins-config-volume" pod "b4a9af3f-b527-443c-8caa-8104a4052805" (UID: "b4a9af3f-b527-443c-8caa-8104a4052805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:44.653702 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.653677 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a9af3f-b527-443c-8caa-8104a4052805-kube-api-access-l4t27" pod "b4a9af3f-b527-443c-8caa-8104a4052805" (UID: "b4a9af3f-b527-443c-8caa-8104a4052805"). InnerVolumeSpecName "kube-api-access-l4t27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:44.752556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.752518 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a9af3f-b527-443c-8caa-8104a4052805-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:44.752556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:44.752552 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4t27\" (UniqueName: \"kubernetes.io/projected/b4a9af3f-b527-443c-8caa-8104a4052805-kube-api-access-l4t27\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:45.582954 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:45.582920 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl" Nov 03 23:18:45.627600 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:45.627561 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl"] Nov 03 23:18:45.637716 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:45.637680 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-22c413-578cb96cchrmtl"] Nov 03 23:18:45.731192 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:45.731155 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a9af3f-b527-443c-8caa-8104a4052805" path="/var/lib/kubelet/pods/b4a9af3f-b527-443c-8caa-8104a4052805/volumes" Nov 03 23:18:47.618251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:47.618208 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5"] Nov 03 23:18:47.618790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:47.618505 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" containerID="cri-o://af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890" gracePeriod=30 Nov 03 23:18:48.383902 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:18:48.383862 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:18:48.692530 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.692493 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" podUID="04f286c9-c002-46a0-b24e-86a51cd21ddd" containerName="main" containerID="cri-o://219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c" gracePeriod=45 Nov 03 23:18:48.941415 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.941386 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:18:48.992485 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.992411 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f286c9-c002-46a0-b24e-86a51cd21ddd-tls-certs\" (UniqueName: \"kubernetes.io/secret/04f286c9-c002-46a0-b24e-86a51cd21ddd-tls-certs\") pod \"04f286c9-c002-46a0-b24e-86a51cd21ddd\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " Nov 03 23:18:48.992485 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.992450 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-home\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-home\") pod \"04f286c9-c002-46a0-b24e-86a51cd21ddd\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " Nov 03 23:18:48.992485 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.992472 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-tmp-dir\") pod \"04f286c9-c002-46a0-b24e-86a51cd21ddd\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " Nov 03 23:18:48.992790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.992567 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-model-cache\") pod \"04f286c9-c002-46a0-b24e-86a51cd21ddd\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " Nov 03 23:18:48.992790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.992611 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f286c9-c002-46a0-b24e-86a51cd21ddd-kube-api-access-8fxvk\" (UniqueName: \"kubernetes.io/projected/04f286c9-c002-46a0-b24e-86a51cd21ddd-kube-api-access-8fxvk\") pod \"04f286c9-c002-46a0-b24e-86a51cd21ddd\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " Nov 03 23:18:48.992790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.992647 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-dshm\") pod \"04f286c9-c002-46a0-b24e-86a51cd21ddd\" (UID: \"04f286c9-c002-46a0-b24e-86a51cd21ddd\") " Nov 03 23:18:48.992790 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.992742 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-home" pod "04f286c9-c002-46a0-b24e-86a51cd21ddd" (UID: "04f286c9-c002-46a0-b24e-86a51cd21ddd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:48.993024 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.992781 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-model-cache" pod "04f286c9-c002-46a0-b24e-86a51cd21ddd" (UID: "04f286c9-c002-46a0-b24e-86a51cd21ddd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:48.993024 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.992827 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-tmp-dir" pod "04f286c9-c002-46a0-b24e-86a51cd21ddd" (UID: "04f286c9-c002-46a0-b24e-86a51cd21ddd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:48.993144 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.993049 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:48.993144 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.993071 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:48.993144 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.993084 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:48.995399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.995341 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-dshm" pod "04f286c9-c002-46a0-b24e-86a51cd21ddd" (UID: "04f286c9-c002-46a0-b24e-86a51cd21ddd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:48.995597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.995426 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f286c9-c002-46a0-b24e-86a51cd21ddd-tls-certs" pod "04f286c9-c002-46a0-b24e-86a51cd21ddd" (UID: "04f286c9-c002-46a0-b24e-86a51cd21ddd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:18:48.995597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:48.995460 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f286c9-c002-46a0-b24e-86a51cd21ddd-kube-api-access-8fxvk" pod "04f286c9-c002-46a0-b24e-86a51cd21ddd" (UID: "04f286c9-c002-46a0-b24e-86a51cd21ddd"). InnerVolumeSpecName "kube-api-access-8fxvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:49.098324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.094433 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04f286c9-c002-46a0-b24e-86a51cd21ddd-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:49.098324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.094474 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxvk\" (UniqueName: \"kubernetes.io/projected/04f286c9-c002-46a0-b24e-86a51cd21ddd-kube-api-access-8fxvk\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:49.098324 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.094503 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04f286c9-c002-46a0-b24e-86a51cd21ddd-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:49.604628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.604587 4903 generic.go:356] "Generic (PLEG): container finished" podID="04f286c9-c002-46a0-b24e-86a51cd21ddd" containerID="219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c" exitCode=0 Nov 03 23:18:49.604810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.604652 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" Nov 03 23:18:49.604810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.604679 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" event={"ID":"04f286c9-c002-46a0-b24e-86a51cd21ddd","Type":"ContainerDied","Data":"219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c"} Nov 03 23:18:49.604810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.604734 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl" event={"ID":"04f286c9-c002-46a0-b24e-86a51cd21ddd","Type":"ContainerDied","Data":"8fcd8d8538ecb14809f6013ec5a619e44e686cafd1e292e44f2ba6b40c1665d6"} Nov 03 23:18:49.604810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.604752 4903 scope.go:122] "RemoveContainer" containerID="219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c" Nov 03 23:18:49.614835 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.614805 4903 scope.go:122] "RemoveContainer" containerID="219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c" Nov 03 23:18:49.615101 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:18:49.615078 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c\": container with ID starting with 219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c not found: ID does not exist" containerID="219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c" Nov 03 23:18:49.615154 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.615111 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c"} err="failed to get container status \"219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c\": rpc error: code = NotFound desc = could not find container \"219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c\": container with ID starting with 219a8244d56d3d9715eb4e3c314c4414c7c56fb86f51821bde1a89d27eaef38c not found: ID does not exist" Nov 03 23:18:49.649844 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.649773 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl"] Nov 03 23:18:49.659656 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.659629 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-22c413/simulator-single-main-22c413-kserve-84486f75d5-4z9xl"] Nov 03 23:18:49.732457 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:49.732414 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f286c9-c002-46a0-b24e-86a51cd21ddd" path="/var/lib/kubelet/pods/04f286c9-c002-46a0-b24e-86a51cd21ddd/volumes" Nov 03 23:18:52.862884 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.862851 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x"] Nov 03 23:18:52.863660 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.863558 4903 cpu_manager.go:414] "containerMap: removing container" podUID="dc97aa8c-9ce6-469b-8ed2-376a731efb6c" containerName="revoke-keys" Nov 03 23:18:52.863660 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.863581 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc97aa8c-9ce6-469b-8ed2-376a731efb6c" containerName="revoke-keys" Nov 03 23:18:52.863660 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.863609 4903 cpu_manager.go:414] "containerMap: removing container" podUID="b4a9af3f-b527-443c-8caa-8104a4052805" containerName="payload-processing" Nov 03 23:18:52.863660 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.863618 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a9af3f-b527-443c-8caa-8104a4052805" containerName="payload-processing" Nov 03 23:18:52.863660 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.863648 4903 cpu_manager.go:414] "containerMap: removing container" podUID="04f286c9-c002-46a0-b24e-86a51cd21ddd" containerName="main" Nov 03 23:18:52.863660 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.863657 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f286c9-c002-46a0-b24e-86a51cd21ddd" containerName="main" Nov 03 23:18:52.864073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.863760 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc97aa8c-9ce6-469b-8ed2-376a731efb6c" containerName="revoke-keys" Nov 03 23:18:52.864073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.863773 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a9af3f-b527-443c-8caa-8104a4052805" containerName="payload-processing" Nov 03 23:18:52.864073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.863786 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f286c9-c002-46a0-b24e-86a51cd21ddd" containerName="main" Nov 03 23:18:52.867336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.867314 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:52.880820 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.880769 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:18:52.890903 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.890874 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ed6ec1b2-kserve-self-signed-certs\"" Nov 03 23:18:52.892632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.892611 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x"] Nov 03 23:18:52.930642 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.930610 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-kube-api-access-26m7t\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-kube-api-access-26m7t\") pod \"9279d7e7-d221-4df4-878b-2e5d92ae6490\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " Nov 03 23:18:52.930815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.930652 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9279d7e7-d221-4df4-878b-2e5d92ae6490-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9279d7e7-d221-4df4-878b-2e5d92ae6490-istiod-ca-cert\") pod \"9279d7e7-d221-4df4-878b-2e5d92ae6490\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " Nov 03 23:18:52.930815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.930675 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-podinfo\") pod \"9279d7e7-d221-4df4-878b-2e5d92ae6490\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " Nov 03 23:18:52.930815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.930720 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-credential-socket\") pod \"9279d7e7-d221-4df4-878b-2e5d92ae6490\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " Nov 03 23:18:52.930815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.930741 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-certs\") pod \"9279d7e7-d221-4df4-878b-2e5d92ae6490\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " Nov 03 23:18:52.930815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.930763 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-socket\") pod \"9279d7e7-d221-4df4-878b-2e5d92ae6490\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " Nov 03 23:18:52.930815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.930805 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-data\") pod \"9279d7e7-d221-4df4-878b-2e5d92ae6490\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " Nov 03 23:18:52.931177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.930838 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-envoy\") pod \"9279d7e7-d221-4df4-878b-2e5d92ae6490\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " Nov 03 23:18:52.931177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.930880 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-token\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-token\") pod \"9279d7e7-d221-4df4-878b-2e5d92ae6490\" (UID: \"9279d7e7-d221-4df4-878b-2e5d92ae6490\") " Nov 03 23:18:52.931177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931028 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-credential-socket" pod "9279d7e7-d221-4df4-878b-2e5d92ae6490" (UID: "9279d7e7-d221-4df4-878b-2e5d92ae6490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:52.931177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931038 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-model-cache\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:52.931177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931066 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-certs" pod "9279d7e7-d221-4df4-878b-2e5d92ae6490" (UID: "9279d7e7-d221-4df4-878b-2e5d92ae6490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:52.931177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931111 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-tmp-dir\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:52.931177 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931116 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9279d7e7-d221-4df4-878b-2e5d92ae6490-istiod-ca-cert" pod "9279d7e7-d221-4df4-878b-2e5d92ae6490" (UID: "9279d7e7-d221-4df4-878b-2e5d92ae6490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:18:52.931611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931226 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-data" pod "9279d7e7-d221-4df4-878b-2e5d92ae6490" (UID: "9279d7e7-d221-4df4-878b-2e5d92ae6490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:52.931611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931341 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-dshm\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:52.931611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931373 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-socket" pod "9279d7e7-d221-4df4-878b-2e5d92ae6490" (UID: "9279d7e7-d221-4df4-878b-2e5d92ae6490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:52.931611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931377 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-home\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:52.931611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931519 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827d71ad-248f-4364-9db3-43d28d36cee1-tls-certs\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:52.931611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931566 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b2ln\" (UniqueName: \"kubernetes.io/projected/827d71ad-248f-4364-9db3-43d28d36cee1-kube-api-access-6b2ln\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:52.931979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931646 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9279d7e7-d221-4df4-878b-2e5d92ae6490-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:52.931979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931661 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:52.931979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931703 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:52.931979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931727 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:52.931979 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.931742 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:52.933082 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.933053 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-envoy" pod "9279d7e7-d221-4df4-878b-2e5d92ae6490" (UID: "9279d7e7-d221-4df4-878b-2e5d92ae6490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:18:52.933181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.933106 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-podinfo" pod "9279d7e7-d221-4df4-878b-2e5d92ae6490" (UID: "9279d7e7-d221-4df4-878b-2e5d92ae6490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:18:52.933181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.933145 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-kube-api-access-26m7t" pod "9279d7e7-d221-4df4-878b-2e5d92ae6490" (UID: "9279d7e7-d221-4df4-878b-2e5d92ae6490"). InnerVolumeSpecName "kube-api-access-26m7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:52.933257 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:52.933198 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-token" pod "9279d7e7-d221-4df4-878b-2e5d92ae6490" (UID: "9279d7e7-d221-4df4-878b-2e5d92ae6490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:18:53.033082 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033027 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-home\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.033280 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033154 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827d71ad-248f-4364-9db3-43d28d36cee1-tls-certs\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.033280 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033202 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b2ln\" (UniqueName: \"kubernetes.io/projected/827d71ad-248f-4364-9db3-43d28d36cee1-kube-api-access-6b2ln\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.033280 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033272 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-model-cache\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.033530 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033319 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-tmp-dir\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.033530 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033369 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-dshm\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.033530 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033435 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:53.033530 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033451 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:53.033530 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033467 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26m7t\" (UniqueName: \"kubernetes.io/projected/9279d7e7-d221-4df4-878b-2e5d92ae6490-kube-api-access-26m7t\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:53.033530 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033480 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9279d7e7-d221-4df4-878b-2e5d92ae6490-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:18:53.033881 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033555 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-home\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.033948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.033873 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-model-cache\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.034239 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.034102 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-tmp-dir\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.036363 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.036280 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-dshm\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.036543 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.036511 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827d71ad-248f-4364-9db3-43d28d36cee1-tls-certs\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.054509 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.054488 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b2ln\" (UniqueName: \"kubernetes.io/projected/827d71ad-248f-4364-9db3-43d28d36cee1-kube-api-access-6b2ln\") pod \"e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.188749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.188714 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:53.488180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.488107 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x"] Nov 03 23:18:53.492015 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:18:53.491987 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod827d71ad_248f_4364_9db3_43d28d36cee1.slice/crio-f7136bf0a7db7348f354651d70c18c57da4fd135cce541dac2a336842d7dae87 WatchSource:0}: Error finding container f7136bf0a7db7348f354651d70c18c57da4fd135cce541dac2a336842d7dae87: Status 404 returned error can't find the container with id f7136bf0a7db7348f354651d70c18c57da4fd135cce541dac2a336842d7dae87 Nov 03 23:18:53.626511 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.626476 4903 generic.go:356] "Generic (PLEG): container finished" podID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerID="af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890" exitCode=0 Nov 03 23:18:53.626682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.626544 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" Nov 03 23:18:53.626682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.626563 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" event={"ID":"9279d7e7-d221-4df4-878b-2e5d92ae6490","Type":"ContainerDied","Data":"af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890"} Nov 03 23:18:53.626682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.626632 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5" event={"ID":"9279d7e7-d221-4df4-878b-2e5d92ae6490","Type":"ContainerDied","Data":"1c6c7fd96cc7b0c75f04749388967c91734ebd777bf237947f884324f2c4c7aa"} Nov 03 23:18:53.626682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.626661 4903 scope.go:122] "RemoveContainer" containerID="af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890" Nov 03 23:18:53.628382 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.628349 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" event={"ID":"827d71ad-248f-4364-9db3-43d28d36cee1","Type":"ContainerStarted","Data":"6fc3534fbdc50f24390b02fb526b86aa5c7510cafc626cb0c926c18f76317b7d"} Nov 03 23:18:53.628491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.628391 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" event={"ID":"827d71ad-248f-4364-9db3-43d28d36cee1","Type":"ContainerStarted","Data":"f7136bf0a7db7348f354651d70c18c57da4fd135cce541dac2a336842d7dae87"} Nov 03 23:18:53.635831 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.635810 4903 scope.go:122] "RemoveContainer" containerID="c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e" Nov 03 23:18:53.651563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.651541 4903 scope.go:122] "RemoveContainer" containerID="af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890" Nov 03 23:18:53.651813 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:18:53.651795 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890\": container with ID starting with af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890 not found: ID does not exist" containerID="af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890" Nov 03 23:18:53.651862 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.651823 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890"} err="failed to get container status \"af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890\": rpc error: code = NotFound desc = could not find container \"af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890\": container with ID starting with af4d8888648c249ea7749c31191530b714b3bf7b3cbf9eaf3b3cf0fbc9193890 not found: ID does not exist" Nov 03 23:18:53.651862 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.651845 4903 scope.go:122] "RemoveContainer" containerID="c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e" Nov 03 23:18:53.652088 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:18:53.652073 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e\": container with ID starting with c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e not found: ID does not exist" containerID="c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e" Nov 03 23:18:53.652141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.652091 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e"} err="failed to get container status \"c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e\": rpc error: code = NotFound desc = could not find container \"c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e\": container with ID starting with c67222ba9166940b47470dee31c6354caa381e7bc28822dc10b09591c628ef5e not found: ID does not exist" Nov 03 23:18:53.678053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.678006 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" podStartSLOduration=1.6779955709999999 podStartE2EDuration="1.677995571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:18:53.676715105 +0000 UTC m=+3960.080656081" watchObservedRunningTime="2026-11-03 23:18:53.677995571 +0000 UTC m=+3960.081936545" Nov 03 23:18:53.709625 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.709602 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5"] Nov 03 23:18:53.714630 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.714610 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-22c413-openshift-default-685f6fd8d9qs4x5"] Nov 03 23:18:53.739788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:53.739717 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" path="/var/lib/kubelet/pods/9279d7e7-d221-4df4-878b-2e5d92ae6490/volumes" Nov 03 23:18:54.189262 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:54.189222 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:18:55.186962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:18:55.186923 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:19:03.189582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:03.189536 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:19:03.202541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:03.202511 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:19:03.738756 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:03.738723 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:19:10.348955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.348915 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf"] Nov 03 23:19:10.349541 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.349521 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" Nov 03 23:19:10.349628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.349543 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" Nov 03 23:19:10.349718 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.349704 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" Nov 03 23:19:10.353949 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.353925 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.371525 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.371491 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf"] Nov 03 23:19:10.379671 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.379636 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ed6ec1b2-kserve-self-signed-certs\"" Nov 03 23:19:10.433628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.433586 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tmp-dir\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.433812 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.433651 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7j4n\" (UniqueName: \"kubernetes.io/projected/7310c1a2-47b8-44c2-9c40-c145251f8ff5-kube-api-access-m7j4n\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.433812 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.433737 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-model-cache\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.433812 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.433794 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tls-certs\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.433932 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.433883 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-home\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.433970 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.433932 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-dshm\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.535184 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.535143 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7j4n\" (UniqueName: \"kubernetes.io/projected/7310c1a2-47b8-44c2-9c40-c145251f8ff5-kube-api-access-m7j4n\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.535184 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.535183 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-model-cache\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.535551 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.535233 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tls-certs\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.535551 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.535268 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-home\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.535551 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.535346 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-dshm\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.535551 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.535396 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tmp-dir\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.535773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.535710 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-model-cache\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.535773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.535738 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-home\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.535773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.535759 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tmp-dir\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.537683 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.537663 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-dshm\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.538063 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.538040 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tls-certs\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.557562 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.557517 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7j4n\" (UniqueName: \"kubernetes.io/projected/7310c1a2-47b8-44c2-9c40-c145251f8ff5-kube-api-access-m7j4n\") pod \"e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.668077 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.668042 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:10.970741 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:10.970714 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf"] Nov 03 23:19:10.974282 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:19:10.974248 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7310c1a2_47b8_44c2_9c40_c145251f8ff5.slice/crio-57206fef625a252d0ef385cfa0afde485fc70ba2839cb13b1f4cbc47962f07fa WatchSource:0}: Error finding container 57206fef625a252d0ef385cfa0afde485fc70ba2839cb13b1f4cbc47962f07fa: Status 404 returned error can't find the container with id 57206fef625a252d0ef385cfa0afde485fc70ba2839cb13b1f4cbc47962f07fa Nov 03 23:19:11.765632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:11.765595 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" event={"ID":"7310c1a2-47b8-44c2-9c40-c145251f8ff5","Type":"ContainerStarted","Data":"c9f89170d8564ae388916ec0b761dcfc25fd90834f6fadf3c8396fb038c17d18"} Nov 03 23:19:11.766048 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:11.765647 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" event={"ID":"7310c1a2-47b8-44c2-9c40-c145251f8ff5","Type":"ContainerStarted","Data":"57206fef625a252d0ef385cfa0afde485fc70ba2839cb13b1f4cbc47962f07fa"} Nov 03 23:19:11.803927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:11.803877 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" podStartSLOduration=1.8038603420000001 podStartE2EDuration="1.803860342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:19:11.802236882 +0000 UTC m=+3978.206177859" watchObservedRunningTime="2026-11-03 23:19:11.803860342 +0000 UTC m=+3978.207801315" Nov 03 23:19:12.668675 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:12.668636 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:12.891851 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:12.891814 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:19:18.383763 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:19:18.383723 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:19:20.668428 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:20.668387 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:20.681178 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:20.681149 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:20.855672 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:20.855639 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:32.011222 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:32.011189 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:19:35.513871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:35.513831 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:19:35.514701 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:35.514659 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:19:36.027630 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:36.027555 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2"] Nov 03 23:19:36.037626 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:36.037593 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07cbd2ec-bpxc2"] Nov 03 23:19:37.731390 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:37.731356 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfcc126-780f-4692-a0d9-c4b18197d1eb" path="/var/lib/kubelet/pods/fbfcc126-780f-4692-a0d9-c4b18197d1eb/volumes" Nov 03 23:19:41.881306 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:41.881248 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x"] Nov 03 23:19:42.858814 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:42.858774 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf"] Nov 03 23:19:44.208137 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:44.208098 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:19:48.383399 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:19:48.383359 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:19:50.515715 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:50.515455 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:19:50.515715 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:50.515562 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:19:50.757317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:50.755371 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-f7pll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:19:50.757317 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:50.755438 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-f7pll" podUID="8e3a38ee-6f79-46eb-a285-8052c4b0228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:19:50.943239 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:50.943188 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:19:56.901186 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:56.901142 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" podUID="827d71ad-248f-4364-9db3-43d28d36cee1" containerName="main" containerID="cri-o://6fc3534fbdc50f24390b02fb526b86aa5c7510cafc626cb0c926c18f76317b7d" gracePeriod=45 Nov 03 23:19:57.064863 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.064826 4903 generic.go:356] "Generic (PLEG): container finished" podID="827d71ad-248f-4364-9db3-43d28d36cee1" containerID="6fc3534fbdc50f24390b02fb526b86aa5c7510cafc626cb0c926c18f76317b7d" exitCode=0 Nov 03 23:19:57.065034 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.064881 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" event={"ID":"827d71ad-248f-4364-9db3-43d28d36cee1","Type":"ContainerDied","Data":"6fc3534fbdc50f24390b02fb526b86aa5c7510cafc626cb0c926c18f76317b7d"} Nov 03 23:19:57.151518 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.151456 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:19:57.290272 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290239 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-tmp-dir\") pod \"827d71ad-248f-4364-9db3-43d28d36cee1\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " Nov 03 23:19:57.290272 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290280 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827d71ad-248f-4364-9db3-43d28d36cee1-kube-api-access-6b2ln\" (UniqueName: \"kubernetes.io/projected/827d71ad-248f-4364-9db3-43d28d36cee1-kube-api-access-6b2ln\") pod \"827d71ad-248f-4364-9db3-43d28d36cee1\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " Nov 03 23:19:57.290519 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290325 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-home\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-home\") pod \"827d71ad-248f-4364-9db3-43d28d36cee1\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " Nov 03 23:19:57.290519 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290353 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-model-cache\") pod \"827d71ad-248f-4364-9db3-43d28d36cee1\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " Nov 03 23:19:57.290519 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290442 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-dshm\") pod \"827d71ad-248f-4364-9db3-43d28d36cee1\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " Nov 03 23:19:57.290519 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290487 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/827d71ad-248f-4364-9db3-43d28d36cee1-tls-certs\" (UniqueName: \"kubernetes.io/secret/827d71ad-248f-4364-9db3-43d28d36cee1-tls-certs\") pod \"827d71ad-248f-4364-9db3-43d28d36cee1\" (UID: \"827d71ad-248f-4364-9db3-43d28d36cee1\") " Nov 03 23:19:57.290767 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290597 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-tmp-dir" pod "827d71ad-248f-4364-9db3-43d28d36cee1" (UID: "827d71ad-248f-4364-9db3-43d28d36cee1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:19:57.290767 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290611 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-home" pod "827d71ad-248f-4364-9db3-43d28d36cee1" (UID: "827d71ad-248f-4364-9db3-43d28d36cee1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:19:57.290767 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290724 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-model-cache" pod "827d71ad-248f-4364-9db3-43d28d36cee1" (UID: "827d71ad-248f-4364-9db3-43d28d36cee1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:19:57.290920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290843 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:57.290920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290861 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:57.290920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.290874 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:57.292807 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.292785 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827d71ad-248f-4364-9db3-43d28d36cee1-tls-certs" pod "827d71ad-248f-4364-9db3-43d28d36cee1" (UID: "827d71ad-248f-4364-9db3-43d28d36cee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:19:57.292961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.292940 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-dshm" pod "827d71ad-248f-4364-9db3-43d28d36cee1" (UID: "827d71ad-248f-4364-9db3-43d28d36cee1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:19:57.293028 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.292957 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827d71ad-248f-4364-9db3-43d28d36cee1-kube-api-access-6b2ln" pod "827d71ad-248f-4364-9db3-43d28d36cee1" (UID: "827d71ad-248f-4364-9db3-43d28d36cee1"). InnerVolumeSpecName "kube-api-access-6b2ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:19:57.391343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.391308 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b2ln\" (UniqueName: \"kubernetes.io/projected/827d71ad-248f-4364-9db3-43d28d36cee1-kube-api-access-6b2ln\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:57.391343 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.391342 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827d71ad-248f-4364-9db3-43d28d36cee1-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:57.391563 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.391356 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827d71ad-248f-4364-9db3-43d28d36cee1-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:57.879997 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:57.879938 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" podUID="7310c1a2-47b8-44c2-9c40-c145251f8ff5" containerName="main" containerID="cri-o://c9f89170d8564ae388916ec0b761dcfc25fd90834f6fadf3c8396fb038c17d18" gracePeriod=45 Nov 03 23:19:58.072714 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.072684 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" Nov 03 23:19:58.073191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.072679 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x" event={"ID":"827d71ad-248f-4364-9db3-43d28d36cee1","Type":"ContainerDied","Data":"f7136bf0a7db7348f354651d70c18c57da4fd135cce541dac2a336842d7dae87"} Nov 03 23:19:58.073191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.072872 4903 scope.go:122] "RemoveContainer" containerID="6fc3534fbdc50f24390b02fb526b86aa5c7510cafc626cb0c926c18f76317b7d" Nov 03 23:19:58.075755 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.075735 4903 generic.go:356] "Generic (PLEG): container finished" podID="7310c1a2-47b8-44c2-9c40-c145251f8ff5" containerID="c9f89170d8564ae388916ec0b761dcfc25fd90834f6fadf3c8396fb038c17d18" exitCode=0 Nov 03 23:19:58.075861 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.075776 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" event={"ID":"7310c1a2-47b8-44c2-9c40-c145251f8ff5","Type":"ContainerDied","Data":"c9f89170d8564ae388916ec0b761dcfc25fd90834f6fadf3c8396fb038c17d18"} Nov 03 23:19:58.117913 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.117888 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x"] Nov 03 23:19:58.128321 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.128265 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ed6ec1b2-kserve-8cddfb7d5-czz5x"] Nov 03 23:19:58.132323 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.132287 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:58.197788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.197756 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tmp-dir\") pod \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " Nov 03 23:19:58.197788 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.197790 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tls-certs\" (UniqueName: \"kubernetes.io/secret/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tls-certs\") pod \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " Nov 03 23:19:58.198039 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.197809 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-dshm\") pod \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " Nov 03 23:19:58.198039 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.197837 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-home\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-home\") pod \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " Nov 03 23:19:58.198039 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.197898 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7310c1a2-47b8-44c2-9c40-c145251f8ff5-kube-api-access-m7j4n\" (UniqueName: \"kubernetes.io/projected/7310c1a2-47b8-44c2-9c40-c145251f8ff5-kube-api-access-m7j4n\") pod \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " Nov 03 23:19:58.198039 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.197933 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-model-cache\") pod \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\" (UID: \"7310c1a2-47b8-44c2-9c40-c145251f8ff5\") " Nov 03 23:19:58.198248 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.198097 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tmp-dir" pod "7310c1a2-47b8-44c2-9c40-c145251f8ff5" (UID: "7310c1a2-47b8-44c2-9c40-c145251f8ff5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:19:58.198248 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.198112 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-home" pod "7310c1a2-47b8-44c2-9c40-c145251f8ff5" (UID: "7310c1a2-47b8-44c2-9c40-c145251f8ff5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:19:58.198345 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.198284 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-model-cache" pod "7310c1a2-47b8-44c2-9c40-c145251f8ff5" (UID: "7310c1a2-47b8-44c2-9c40-c145251f8ff5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:19:58.198345 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.198336 4903 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-home\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:58.198432 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.198357 4903 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tmp-dir\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:58.200035 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.200007 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-dshm" pod "7310c1a2-47b8-44c2-9c40-c145251f8ff5" (UID: "7310c1a2-47b8-44c2-9c40-c145251f8ff5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:19:58.200161 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.200057 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tls-certs" pod "7310c1a2-47b8-44c2-9c40-c145251f8ff5" (UID: "7310c1a2-47b8-44c2-9c40-c145251f8ff5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:19:58.200227 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.200196 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7310c1a2-47b8-44c2-9c40-c145251f8ff5-kube-api-access-m7j4n" pod "7310c1a2-47b8-44c2-9c40-c145251f8ff5" (UID: "7310c1a2-47b8-44c2-9c40-c145251f8ff5"). InnerVolumeSpecName "kube-api-access-m7j4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:19:58.230641 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.230601 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:19:58.298966 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.298886 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7j4n\" (UniqueName: \"kubernetes.io/projected/7310c1a2-47b8-44c2-9c40-c145251f8ff5-kube-api-access-m7j4n\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:58.298966 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.298918 4903 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-model-cache\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:58.298966 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.298929 4903 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7310c1a2-47b8-44c2-9c40-c145251f8ff5-tls-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:58.298966 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:58.298939 4903 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7310c1a2-47b8-44c2-9c40-c145251f8ff5-dshm\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:19:59.081916 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:59.081883 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" Nov 03 23:19:59.082364 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:59.081892 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf" event={"ID":"7310c1a2-47b8-44c2-9c40-c145251f8ff5","Type":"ContainerDied","Data":"57206fef625a252d0ef385cfa0afde485fc70ba2839cb13b1f4cbc47962f07fa"} Nov 03 23:19:59.082364 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:59.081999 4903 scope.go:122] "RemoveContainer" containerID="c9f89170d8564ae388916ec0b761dcfc25fd90834f6fadf3c8396fb038c17d18" Nov 03 23:19:59.126597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:59.126572 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf"] Nov 03 23:19:59.137517 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:59.137486 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ed6ec1b2-kserve-5b8495bb45-6vwvf"] Nov 03 23:19:59.732569 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:59.732532 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7310c1a2-47b8-44c2-9c40-c145251f8ff5" path="/var/lib/kubelet/pods/7310c1a2-47b8-44c2-9c40-c145251f8ff5/volumes" Nov 03 23:19:59.733056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:19:59.733033 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827d71ad-248f-4364-9db3-43d28d36cee1" path="/var/lib/kubelet/pods/827d71ad-248f-4364-9db3-43d28d36cee1/volumes" Nov 03 23:20:02.794923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:02.794883 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:20:05.515523 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:05.515482 4903 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fkpql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:20:05.516158 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:05.516122 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fkpql" podUID="08263e96-b3ee-4661-9110-4a06dcae57e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:20:08.027005 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:08.026968 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv"] Nov 03 23:20:08.037200 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:08.037170 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n8bgv"] Nov 03 23:20:09.731967 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:09.731931 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2feee52-65e8-46bc-9d40-3e3e0ee23e68" path="/var/lib/kubelet/pods/a2feee52-65e8-46bc-9d40-3e3e0ee23e68/volumes" Nov 03 23:20:18.382559 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:20:18.382515 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:20:20.609400 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:20.609358 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:20:29.014910 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.014868 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2"] Nov 03 23:20:29.015655 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.015628 4903 cpu_manager.go:414] "containerMap: removing container" podUID="7310c1a2-47b8-44c2-9c40-c145251f8ff5" containerName="main" Nov 03 23:20:29.015715 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.015662 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7310c1a2-47b8-44c2-9c40-c145251f8ff5" containerName="main" Nov 03 23:20:29.015766 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.015718 4903 cpu_manager.go:414] "containerMap: removing container" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" Nov 03 23:20:29.015766 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.015731 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" Nov 03 23:20:29.015766 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.015757 4903 cpu_manager.go:414] "containerMap: removing container" podUID="827d71ad-248f-4364-9db3-43d28d36cee1" containerName="main" Nov 03 23:20:29.015906 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.015767 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827d71ad-248f-4364-9db3-43d28d36cee1" containerName="main" Nov 03 23:20:29.015906 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.015896 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="827d71ad-248f-4364-9db3-43d28d36cee1" containerName="main" Nov 03 23:20:29.016010 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.015918 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="7310c1a2-47b8-44c2-9c40-c145251f8ff5" containerName="main" Nov 03 23:20:29.016010 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.015936 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="9279d7e7-d221-4df4-878b-2e5d92ae6490" containerName="istio-proxy" Nov 03 23:20:29.020164 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.020140 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.038224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.038193 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a6ecd9bf-openshift-default-dockercfg-5q26h\"" Nov 03 23:20:29.050015 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.049984 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2"] Nov 03 23:20:29.116759 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.116726 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-podinfo\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.116924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.116774 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-token\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.116924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.116795 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe76cc1-30c2-4089-aa13-b95912955e9d-istiod-ca-cert\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.116924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.116880 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-socket\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.116924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.116914 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kstdw\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-kube-api-access-kstdw\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.117100 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.116983 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-credential-socket\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.117100 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.117038 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-data\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.117100 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.117080 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-envoy\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.117203 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.117134 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-certs\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.217877 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.217844 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-data\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218085 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.217891 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-envoy\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218085 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218010 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-certs\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218085 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218053 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-podinfo\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218098 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-token\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218132 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe76cc1-30c2-4089-aa13-b95912955e9d-istiod-ca-cert\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218175 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-socket\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218274 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218210 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kstdw\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-kube-api-access-kstdw\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218511 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218287 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-credential-socket\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218511 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218337 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-data\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218511 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218413 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-certs\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218594 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-socket\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218678 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218641 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-credential-socket\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.218787 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.218766 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe76cc1-30c2-4089-aa13-b95912955e9d-istiod-ca-cert\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.220382 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.220360 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-envoy\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.220897 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.220876 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-podinfo\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.239948 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.239913 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kstdw\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-kube-api-access-kstdw\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.240081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.239917 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-token\") pod \"e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.333769 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.333686 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:29.656258 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.656222 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2"] Nov 03 23:20:29.675649 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:20:29.675614 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe76cc1_30c2_4089_aa13_b95912955e9d.slice/crio-7eda0dbae4e5b24465e5d9a977a74c7f4366bb01608434484feda014686fb38b WatchSource:0}: Error finding container 7eda0dbae4e5b24465e5d9a977a74c7f4366bb01608434484feda014686fb38b: Status 404 returned error can't find the container with id 7eda0dbae4e5b24465e5d9a977a74c7f4366bb01608434484feda014686fb38b Nov 03 23:20:29.678067 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.678029 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:20:29.678190 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.678123 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:20:29.678190 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:29.678167 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:20:30.245587 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:30.245553 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" event={"ID":"afe76cc1-30c2-4089-aa13-b95912955e9d","Type":"ContainerStarted","Data":"331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934"} Nov 03 23:20:30.245976 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:30.245611 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" event={"ID":"afe76cc1-30c2-4089-aa13-b95912955e9d","Type":"ContainerStarted","Data":"7eda0dbae4e5b24465e5d9a977a74c7f4366bb01608434484feda014686fb38b"} Nov 03 23:20:30.298661 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:30.298604 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" podStartSLOduration=2.298589803 podStartE2EDuration="2.298589803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:30.296709105 +0000 UTC m=+4056.700650071" watchObservedRunningTime="2026-11-03 23:20:30.298589803 +0000 UTC m=+4056.702530778" Nov 03 23:20:30.334154 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:30.334116 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:30.339446 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:30.339413 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:31.250457 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:31.250420 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:31.251643 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:31.251621 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:34.504907 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.504868 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn"] Nov 03 23:20:34.508463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.508430 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:34.531518 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.531491 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a6ecd9bf\"" Nov 03 23:20:34.534707 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.534679 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn"] Nov 03 23:20:34.576824 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.576791 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gh7p\" (UniqueName: \"kubernetes.io/projected/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-kube-api-access-6gh7p\") pod \"payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn\" (UID: \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:34.576989 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.576906 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn\" (UID: \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:34.606160 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.606131 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a6ecd9bf-dockercfg-wsnjt\"" Nov 03 23:20:34.618748 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.618722 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk"] Nov 03 23:20:34.622185 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.622164 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:34.640676 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.640654 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk"] Nov 03 23:20:34.677854 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.677812 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn\" (UID: \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:34.678043 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.677913 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gh7p\" (UniqueName: \"kubernetes.io/projected/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-kube-api-access-6gh7p\") pod \"payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn\" (UID: \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:34.678643 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.678624 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn\" (UID: \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:34.700154 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.700117 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gh7p\" (UniqueName: \"kubernetes.io/projected/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-kube-api-access-6gh7p\") pod \"payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn\" (UID: \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:34.779496 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.779396 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60220307-46c7-432b-af25-38f4b066d428-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk\" (UID: \"60220307-46c7-432b-af25-38f4b066d428\") " pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:34.779662 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.779593 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/60220307-46c7-432b-af25-38f4b066d428-kube-api-access-zz2mc\") pod \"payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk\" (UID: \"60220307-46c7-432b-af25-38f4b066d428\") " pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:34.822457 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.822424 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:34.880620 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.880589 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/60220307-46c7-432b-af25-38f4b066d428-kube-api-access-zz2mc\") pod \"payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk\" (UID: \"60220307-46c7-432b-af25-38f4b066d428\") " pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:34.880751 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.880657 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60220307-46c7-432b-af25-38f4b066d428-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk\" (UID: \"60220307-46c7-432b-af25-38f4b066d428\") " pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:34.881390 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.881367 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60220307-46c7-432b-af25-38f4b066d428-plugins-config-volume\") pod \"payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk\" (UID: \"60220307-46c7-432b-af25-38f4b066d428\") " pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:34.907165 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.907130 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/60220307-46c7-432b-af25-38f4b066d428-kube-api-access-zz2mc\") pod \"payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk\" (UID: \"60220307-46c7-432b-af25-38f4b066d428\") " pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:34.934889 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:34.934858 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:35.195330 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:35.195258 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn"] Nov 03 23:20:35.198679 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:20:35.198648 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0801b66c_f7c9_4e03_bc62_1883e4cc5f56.slice/crio-f1ae4b59e8ff6b7bb2090c49b59bfecd4dce8921ca4d94ee3c1b3aa63b17f36a WatchSource:0}: Error finding container f1ae4b59e8ff6b7bb2090c49b59bfecd4dce8921ca4d94ee3c1b3aa63b17f36a: Status 404 returned error can't find the container with id f1ae4b59e8ff6b7bb2090c49b59bfecd4dce8921ca4d94ee3c1b3aa63b17f36a Nov 03 23:20:35.244677 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:35.244654 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk"] Nov 03 23:20:35.258663 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:20:35.258637 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60220307_46c7_432b_af25_38f4b066d428.slice/crio-d99c110e583abf516a73e9edece89d036c4f974d24429f17225693d62e7ea8f5 WatchSource:0}: Error finding container d99c110e583abf516a73e9edece89d036c4f974d24429f17225693d62e7ea8f5: Status 404 returned error can't find the container with id d99c110e583abf516a73e9edece89d036c4f974d24429f17225693d62e7ea8f5 Nov 03 23:20:35.270136 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:35.270110 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" event={"ID":"60220307-46c7-432b-af25-38f4b066d428","Type":"ContainerStarted","Data":"d99c110e583abf516a73e9edece89d036c4f974d24429f17225693d62e7ea8f5"} Nov 03 23:20:35.272121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:35.272102 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" event={"ID":"0801b66c-f7c9-4e03-bc62-1883e4cc5f56","Type":"ContainerStarted","Data":"f1ae4b59e8ff6b7bb2090c49b59bfecd4dce8921ca4d94ee3c1b3aa63b17f36a"} Nov 03 23:20:36.278130 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:36.278093 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" event={"ID":"60220307-46c7-432b-af25-38f4b066d428","Type":"ContainerStarted","Data":"651a8743815ea9c50d46455011a7a4df289de817846c357f34af5e6df2b36477"} Nov 03 23:20:36.278645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:36.278158 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:36.279635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:36.279610 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" event={"ID":"0801b66c-f7c9-4e03-bc62-1883e4cc5f56","Type":"ContainerStarted","Data":"feab9b9ce06883d019bdbab8e6c3949474c4aebad8c6c0a20bf6cf2b821b724d"} Nov 03 23:20:36.279749 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:36.279656 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:36.316052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:36.315997 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" podStartSLOduration=2.315982183 podStartE2EDuration="2.315982183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:36.313826303 +0000 UTC m=+4062.717767278" watchObservedRunningTime="2026-11-03 23:20:36.315982183 +0000 UTC m=+4062.719923196" Nov 03 23:20:36.350119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:36.350071 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" podStartSLOduration=2.350056477 podStartE2EDuration="2.350056477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:20:36.349289179 +0000 UTC m=+4062.753230152" watchObservedRunningTime="2026-11-03 23:20:36.350056477 +0000 UTC m=+4062.753997452" Nov 03 23:20:37.675497 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:37.675462 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:20:47.288088 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.288049 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:20:47.289675 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.288130 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:20:47.291406 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.291385 4903 patch_prober.go:29] interesting pod/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:20:47.291532 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.291447 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:20:47.769794 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.769755 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5"] Nov 03 23:20:47.772833 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.772808 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:47.785617 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.785596 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v4tdh\"" Nov 03 23:20:47.785729 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.785631 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:20:47.785729 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.785658 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:20:47.788265 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.788248 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:20:47.797592 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.797562 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5"] Nov 03 23:20:47.831102 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.831066 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6cpn\" (UniqueName: \"kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:47.831264 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.831135 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:47.831336 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.831317 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:47.932169 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.932131 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:47.932373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.932187 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6cpn\" (UniqueName: \"kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:47.932373 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.932235 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:47.932821 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.932783 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:47.934565 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:47.934544 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:47.956794 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:20:47.956766 4903 projected.go:196] Error preparing data for projected volume kube-api-access-v6cpn for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 23:20:47.956924 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:20:47.956828 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn podName:057459ff-d97f-49fa-a2fd-96a5a19d203b nodeName:}" failed. No retries permitted until 2026-11-03 23:20:48.456812448 +0000 UTC m=+4074.860753401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v6cpn" (UniqueName: "kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn") pod "maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" (UID: "057459ff-d97f-49fa-a2fd-96a5a19d203b") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 23:20:48.339729 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:48.339693 4903 generic.go:356] "Generic (PLEG): container finished" podID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerID="331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934" exitCode=0 Nov 03 23:20:48.340200 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:48.339758 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" event={"ID":"afe76cc1-30c2-4089-aa13-b95912955e9d","Type":"ContainerDied","Data":"331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934"} Nov 03 23:20:48.340200 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:48.340097 4903 scope.go:122] "RemoveContainer" containerID="331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934" Nov 03 23:20:48.341495 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:48.341464 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:20:48.341615 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:48.341549 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:20:48.341615 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:48.341597 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:20:48.384380 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:20:48.384340 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:20:48.539134 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:48.539102 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6cpn\" (UniqueName: \"kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:48.552758 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:20:48.552731 4903 projected.go:196] Error preparing data for projected volume kube-api-access-v6cpn for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 23:20:48.552894 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:20:48.552790 4903 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn podName:057459ff-d97f-49fa-a2fd-96a5a19d203b nodeName:}" failed. No retries permitted until 2026-11-03 23:20:49.552775614 +0000 UTC m=+4075.956716580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v6cpn" (UniqueName: "kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn") pod "maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" (UID: "057459ff-d97f-49fa-a2fd-96a5a19d203b") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 23:20:49.372062 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:49.372028 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" event={"ID":"afe76cc1-30c2-4089-aa13-b95912955e9d","Type":"ContainerStarted","Data":"7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710"} Nov 03 23:20:49.570007 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:49.569959 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6cpn\" (UniqueName: \"kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:49.591510 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:49.591476 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6cpn\" (UniqueName: \"kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn\") pod \"maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:49.884756 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:49.884715 4903 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" secret="" err="secret \"maas-api-cleanup-dockercfg-v4tdh\" not found" Nov 03 23:20:49.884947 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:49.884792 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:50.180418 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:50.180317 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5"] Nov 03 23:20:50.183489 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:20:50.183461 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057459ff_d97f_49fa_a2fd_96a5a19d203b.slice/crio-8030dc663ef2b37b3f928d0ae0d03bb8d6b5277cbaaa652aac43c19634db8f12 WatchSource:0}: Error finding container 8030dc663ef2b37b3f928d0ae0d03bb8d6b5277cbaaa652aac43c19634db8f12: Status 404 returned error can't find the container with id 8030dc663ef2b37b3f928d0ae0d03bb8d6b5277cbaaa652aac43c19634db8f12 Nov 03 23:20:50.333880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:50.333845 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:50.334129 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:50.334108 4903 patch_prober.go:29] interesting pod/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Nov 03 23:20:50.334199 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:50.334154 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Nov 03 23:20:50.378188 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:50.378154 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" event={"ID":"057459ff-d97f-49fa-a2fd-96a5a19d203b","Type":"ContainerStarted","Data":"8030dc663ef2b37b3f928d0ae0d03bb8d6b5277cbaaa652aac43c19634db8f12"} Nov 03 23:20:51.334669 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:51.334629 4903 patch_prober.go:29] interesting pod/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Nov 03 23:20:51.334893 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:51.334693 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Nov 03 23:20:51.385050 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:51.385016 4903 generic.go:356] "Generic (PLEG): container finished" podID="057459ff-d97f-49fa-a2fd-96a5a19d203b" containerID="99abc89987f3f98b28b24bf939304deb10f7ff7756043564fcc1292b9e98bb19" exitCode=0 Nov 03 23:20:51.385489 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:51.385099 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" event={"ID":"057459ff-d97f-49fa-a2fd-96a5a19d203b","Type":"ContainerDied","Data":"99abc89987f3f98b28b24bf939304deb10f7ff7756043564fcc1292b9e98bb19"} Nov 03 23:20:52.335072 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.335033 4903 patch_prober.go:29] interesting pod/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Nov 03 23:20:52.335263 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.335089 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Nov 03 23:20:52.573348 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.573322 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:52.600109 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.600044 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-openshift-ca\") pod \"057459ff-d97f-49fa-a2fd-96a5a19d203b\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " Nov 03 23:20:52.600229 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.600169 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn\" (UniqueName: \"kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn\") pod \"057459ff-d97f-49fa-a2fd-96a5a19d203b\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " Nov 03 23:20:52.600286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.600235 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-xks-ca\") pod \"057459ff-d97f-49fa-a2fd-96a5a19d203b\" (UID: \"057459ff-d97f-49fa-a2fd-96a5a19d203b\") " Nov 03 23:20:52.600460 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.600436 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-openshift-ca" pod "057459ff-d97f-49fa-a2fd-96a5a19d203b" (UID: "057459ff-d97f-49fa-a2fd-96a5a19d203b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:20:52.600597 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.600570 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:20:52.602348 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.602324 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-xks-ca" pod "057459ff-d97f-49fa-a2fd-96a5a19d203b" (UID: "057459ff-d97f-49fa-a2fd-96a5a19d203b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:20:52.602421 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.602346 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn" pod "057459ff-d97f-49fa-a2fd-96a5a19d203b" (UID: "057459ff-d97f-49fa-a2fd-96a5a19d203b"). InnerVolumeSpecName "kube-api-access-v6cpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:20:52.701808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.701768 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057459ff-d97f-49fa-a2fd-96a5a19d203b-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:20:52.701808 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:52.701800 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6cpn\" (UniqueName: \"kubernetes.io/projected/057459ff-d97f-49fa-a2fd-96a5a19d203b-kube-api-access-v6cpn\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:20:53.334745 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:53.334712 4903 patch_prober.go:29] interesting pod/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" start-of-body= Nov 03 23:20:53.334940 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:53.334761 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.154:15021/healthz/ready\": dial tcp 10.132.1.154:15021: connect: connection refused" Nov 03 23:20:53.395608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:53.395569 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" event={"ID":"057459ff-d97f-49fa-a2fd-96a5a19d203b","Type":"ContainerDied","Data":"8030dc663ef2b37b3f928d0ae0d03bb8d6b5277cbaaa652aac43c19634db8f12"} Nov 03 23:20:53.395809 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:53.395625 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8030dc663ef2b37b3f928d0ae0d03bb8d6b5277cbaaa652aac43c19634db8f12" Nov 03 23:20:53.395809 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:53.395688 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5" Nov 03 23:20:54.341943 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:54.341881 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:54.342471 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:54.342325 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:54.345653 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:54.345629 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:20:58.185516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:58.185477 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk"] Nov 03 23:20:58.216301 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:20:58.216256 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn"] Nov 03 23:21:00.797713 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:00.797678 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5"] Nov 03 23:21:00.808221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:00.808191 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6ecd9bf-jcfq5"] Nov 03 23:21:01.733202 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:01.733163 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057459ff-d97f-49fa-a2fd-96a5a19d203b" path="/var/lib/kubelet/pods/057459ff-d97f-49fa-a2fd-96a5a19d203b/volumes" Nov 03 23:21:01.801138 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:01.801106 4903 scope.go:122] "RemoveContainer" containerID="199b00332a05deeb13f59788edf28bb0f3da9c5c67d32f3ecfb75715066b28c4" Nov 03 23:21:01.809877 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:01.809857 4903 scope.go:122] "RemoveContainer" containerID="cda66b0e1199001af0d0221574b33deeabeafed0132bb9e501f2e2ae3633fbc5" Nov 03 23:21:02.591187 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:02.591142 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:21:03.186818 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.186776 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" podUID="60220307-46c7-432b-af25-38f4b066d428" containerName="payload-processing" containerID="cri-o://651a8743815ea9c50d46455011a7a4df289de817846c357f34af5e6df2b36477" gracePeriod=25 Nov 03 23:21:03.220380 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.220149 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" podUID="0801b66c-f7c9-4e03-bc62-1883e4cc5f56" containerName="payload-pre-processing" containerID="cri-o://feab9b9ce06883d019bdbab8e6c3949474c4aebad8c6c0a20bf6cf2b821b724d" gracePeriod=25 Nov 03 23:21:03.458214 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.458180 4903 generic.go:356] "Generic (PLEG): container finished" podID="0801b66c-f7c9-4e03-bc62-1883e4cc5f56" containerID="feab9b9ce06883d019bdbab8e6c3949474c4aebad8c6c0a20bf6cf2b821b724d" exitCode=0 Nov 03 23:21:03.458445 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.458366 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" event={"ID":"0801b66c-f7c9-4e03-bc62-1883e4cc5f56","Type":"ContainerDied","Data":"feab9b9ce06883d019bdbab8e6c3949474c4aebad8c6c0a20bf6cf2b821b724d"} Nov 03 23:21:03.460389 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.460366 4903 generic.go:356] "Generic (PLEG): container finished" podID="60220307-46c7-432b-af25-38f4b066d428" containerID="651a8743815ea9c50d46455011a7a4df289de817846c357f34af5e6df2b36477" exitCode=0 Nov 03 23:21:03.460522 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.460451 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" event={"ID":"60220307-46c7-432b-af25-38f4b066d428","Type":"ContainerDied","Data":"651a8743815ea9c50d46455011a7a4df289de817846c357f34af5e6df2b36477"} Nov 03 23:21:03.460522 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.460513 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" event={"ID":"60220307-46c7-432b-af25-38f4b066d428","Type":"ContainerDied","Data":"d99c110e583abf516a73e9edece89d036c4f974d24429f17225693d62e7ea8f5"} Nov 03 23:21:03.460653 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.460538 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d99c110e583abf516a73e9edece89d036c4f974d24429f17225693d62e7ea8f5" Nov 03 23:21:03.479211 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.479189 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:21:03.505342 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.505314 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:21:03.607614 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.607577 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60220307-46c7-432b-af25-38f4b066d428-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60220307-46c7-432b-af25-38f4b066d428-plugins-config-volume\") pod \"60220307-46c7-432b-af25-38f4b066d428\" (UID: \"60220307-46c7-432b-af25-38f4b066d428\") " Nov 03 23:21:03.607804 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.607622 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-plugins-config-volume\") pod \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\" (UID: \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\") " Nov 03 23:21:03.607853 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.607801 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-kube-api-access-6gh7p\" (UniqueName: \"kubernetes.io/projected/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-kube-api-access-6gh7p\") pod \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\" (UID: \"0801b66c-f7c9-4e03-bc62-1883e4cc5f56\") " Nov 03 23:21:03.607853 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.607835 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60220307-46c7-432b-af25-38f4b066d428-kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/60220307-46c7-432b-af25-38f4b066d428-kube-api-access-zz2mc\") pod \"60220307-46c7-432b-af25-38f4b066d428\" (UID: \"60220307-46c7-432b-af25-38f4b066d428\") " Nov 03 23:21:03.608001 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.607963 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60220307-46c7-432b-af25-38f4b066d428-plugins-config-volume" pod "60220307-46c7-432b-af25-38f4b066d428" (UID: "60220307-46c7-432b-af25-38f4b066d428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:21:03.608092 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.608074 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-plugins-config-volume" pod "0801b66c-f7c9-4e03-bc62-1883e4cc5f56" (UID: "0801b66c-f7c9-4e03-bc62-1883e4cc5f56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:21:03.608170 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.608155 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:03.608222 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.608171 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60220307-46c7-432b-af25-38f4b066d428-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:03.610083 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.610057 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-kube-api-access-6gh7p" pod "0801b66c-f7c9-4e03-bc62-1883e4cc5f56" (UID: "0801b66c-f7c9-4e03-bc62-1883e4cc5f56"). InnerVolumeSpecName "kube-api-access-6gh7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:21:03.610083 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.610070 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60220307-46c7-432b-af25-38f4b066d428-kube-api-access-zz2mc" pod "60220307-46c7-432b-af25-38f4b066d428" (UID: "60220307-46c7-432b-af25-38f4b066d428"). InnerVolumeSpecName "kube-api-access-zz2mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:21:03.709251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.709164 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gh7p\" (UniqueName: \"kubernetes.io/projected/0801b66c-f7c9-4e03-bc62-1883e4cc5f56-kube-api-access-6gh7p\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:03.709251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:03.709204 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz2mc\" (UniqueName: \"kubernetes.io/projected/60220307-46c7-432b-af25-38f4b066d428-kube-api-access-zz2mc\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:04.031632 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.031545 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2"] Nov 03 23:21:04.031832 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.031752 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" containerID="cri-o://7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710" gracePeriod=30 Nov 03 23:21:04.465923 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.465878 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" event={"ID":"0801b66c-f7c9-4e03-bc62-1883e4cc5f56","Type":"ContainerDied","Data":"f1ae4b59e8ff6b7bb2090c49b59bfecd4dce8921ca4d94ee3c1b3aa63b17f36a"} Nov 03 23:21:04.466398 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.465949 4903 scope.go:122] "RemoveContainer" containerID="feab9b9ce06883d019bdbab8e6c3949474c4aebad8c6c0a20bf6cf2b821b724d" Nov 03 23:21:04.466398 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.465900 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn" Nov 03 23:21:04.466398 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.465905 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk" Nov 03 23:21:04.510463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.510434 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk"] Nov 03 23:21:04.513070 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.513047 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a6ecd9bf-5ffcbfd4b-97clk"] Nov 03 23:21:04.543499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.543468 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn"] Nov 03 23:21:04.549198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:04.549176 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a6ecd9bf-8bb8c667-vfwxn"] Nov 03 23:21:05.735802 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.735760 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0801b66c-f7c9-4e03-bc62-1883e4cc5f56" path="/var/lib/kubelet/pods/0801b66c-f7c9-4e03-bc62-1883e4cc5f56/volumes" Nov 03 23:21:05.736455 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.736427 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60220307-46c7-432b-af25-38f4b066d428" path="/var/lib/kubelet/pods/60220307-46c7-432b-af25-38f4b066d428/volumes" Nov 03 23:21:05.737015 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.736994 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz"] Nov 03 23:21:05.737926 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.737896 4903 cpu_manager.go:414] "containerMap: removing container" podUID="057459ff-d97f-49fa-a2fd-96a5a19d203b" containerName="revoke-keys" Nov 03 23:21:05.738053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.737927 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057459ff-d97f-49fa-a2fd-96a5a19d203b" containerName="revoke-keys" Nov 03 23:21:05.738053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.737969 4903 cpu_manager.go:414] "containerMap: removing container" podUID="60220307-46c7-432b-af25-38f4b066d428" containerName="payload-processing" Nov 03 23:21:05.738053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.737979 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60220307-46c7-432b-af25-38f4b066d428" containerName="payload-processing" Nov 03 23:21:05.738053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.738006 4903 cpu_manager.go:414] "containerMap: removing container" podUID="0801b66c-f7c9-4e03-bc62-1883e4cc5f56" containerName="payload-pre-processing" Nov 03 23:21:05.738053 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.738017 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0801b66c-f7c9-4e03-bc62-1883e4cc5f56" containerName="payload-pre-processing" Nov 03 23:21:05.738356 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.738172 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="60220307-46c7-432b-af25-38f4b066d428" containerName="payload-processing" Nov 03 23:21:05.738356 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.738196 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="0801b66c-f7c9-4e03-bc62-1883e4cc5f56" containerName="payload-pre-processing" Nov 03 23:21:05.738356 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.738215 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="057459ff-d97f-49fa-a2fd-96a5a19d203b" containerName="revoke-keys" Nov 03 23:21:05.742364 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.742341 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.758480 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.758444 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-02e08d-openshift-default-dockercfg-n22cg\"" Nov 03 23:21:05.771286 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.771257 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz"] Nov 03 23:21:05.830469 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.830432 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-socket\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.830635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.830486 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-data\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.830635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.830575 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-credential-socket\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.830635 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.830603 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b97e1d-aa29-4d50-ac25-2236b01aab43-istiod-ca-cert\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.830806 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.830792 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-podinfo\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.830849 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.830831 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-envoy\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.830888 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.830853 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-token\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.830957 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.830944 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-certs\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.831054 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.831032 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbrp\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-kube-api-access-dbbrp\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932211 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932163 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-envoy\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932220 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-token\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932265 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-certs\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932345 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbrp\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-kube-api-access-dbbrp\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932385 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-socket\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932440 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932421 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-data\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932754 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932457 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-credential-socket\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932754 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932488 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b97e1d-aa29-4d50-ac25-2236b01aab43-istiod-ca-cert\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932754 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932547 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-podinfo\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932776 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-certs\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.932927 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.932850 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-data\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.933141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.933116 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-socket\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.933254 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.933198 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-credential-socket\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.933562 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.933538 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b97e1d-aa29-4d50-ac25-2236b01aab43-istiod-ca-cert\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.935692 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.935648 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-envoy\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.935871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.935844 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-podinfo\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.953403 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.953373 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-token\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:05.953629 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:05.953609 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbrp\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-kube-api-access-dbbrp\") pod \"e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:06.054825 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:06.054695 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:06.372220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:06.372195 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz"] Nov 03 23:21:06.374791 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:21:06.374763 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b97e1d_aa29_4d50_ac25_2236b01aab43.slice/crio-4277baba31b3ae9c232b8296fdf2b99ea96bd071dcd2af83a4574eb46089c298 WatchSource:0}: Error finding container 4277baba31b3ae9c232b8296fdf2b99ea96bd071dcd2af83a4574eb46089c298: Status 404 returned error can't find the container with id 4277baba31b3ae9c232b8296fdf2b99ea96bd071dcd2af83a4574eb46089c298 Nov 03 23:21:06.377081 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:06.377048 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:21:06.377181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:06.377118 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:21:06.377181 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:06.377145 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:21:06.480647 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:06.480608 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" event={"ID":"05b97e1d-aa29-4d50-ac25-2236b01aab43","Type":"ContainerStarted","Data":"91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0"} Nov 03 23:21:06.480829 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:06.480675 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" event={"ID":"05b97e1d-aa29-4d50-ac25-2236b01aab43","Type":"ContainerStarted","Data":"4277baba31b3ae9c232b8296fdf2b99ea96bd071dcd2af83a4574eb46089c298"} Nov 03 23:21:06.522318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:06.522240 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" podStartSLOduration=1.522226284 podStartE2EDuration="1.522226284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:21:06.521671261 +0000 UTC m=+4092.925612236" watchObservedRunningTime="2026-11-03 23:21:06.522226284 +0000 UTC m=+4092.926167259" Nov 03 23:21:07.055419 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:07.055383 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:07.060684 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:07.060655 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:07.486377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:07.486346 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:07.487306 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:07.487275 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:09.279880 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.279850 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:21:09.366901 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.366828 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-kube-api-access-kstdw\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-kube-api-access-kstdw\") pod \"afe76cc1-30c2-4089-aa13-b95912955e9d\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " Nov 03 23:21:09.366901 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.366897 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-envoy\") pod \"afe76cc1-30c2-4089-aa13-b95912955e9d\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " Nov 03 23:21:09.367150 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.366924 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe76cc1-30c2-4089-aa13-b95912955e9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe76cc1-30c2-4089-aa13-b95912955e9d-istiod-ca-cert\") pod \"afe76cc1-30c2-4089-aa13-b95912955e9d\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " Nov 03 23:21:09.367150 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367035 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-credential-socket\") pod \"afe76cc1-30c2-4089-aa13-b95912955e9d\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " Nov 03 23:21:09.367150 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367085 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-podinfo\") pod \"afe76cc1-30c2-4089-aa13-b95912955e9d\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " Nov 03 23:21:09.367150 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367118 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-data\") pod \"afe76cc1-30c2-4089-aa13-b95912955e9d\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " Nov 03 23:21:09.367433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367178 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-certs\") pod \"afe76cc1-30c2-4089-aa13-b95912955e9d\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " Nov 03 23:21:09.367433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367325 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-token\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-token\") pod \"afe76cc1-30c2-4089-aa13-b95912955e9d\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " Nov 03 23:21:09.367433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367390 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-socket\") pod \"afe76cc1-30c2-4089-aa13-b95912955e9d\" (UID: \"afe76cc1-30c2-4089-aa13-b95912955e9d\") " Nov 03 23:21:09.367433 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367387 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-credential-socket" pod "afe76cc1-30c2-4089-aa13-b95912955e9d" (UID: "afe76cc1-30c2-4089-aa13-b95912955e9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:21:09.367639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367410 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe76cc1-30c2-4089-aa13-b95912955e9d-istiod-ca-cert" pod "afe76cc1-30c2-4089-aa13-b95912955e9d" (UID: "afe76cc1-30c2-4089-aa13-b95912955e9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:21:09.367639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367472 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-data" pod "afe76cc1-30c2-4089-aa13-b95912955e9d" (UID: "afe76cc1-30c2-4089-aa13-b95912955e9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:21:09.367639 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367479 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-certs" pod "afe76cc1-30c2-4089-aa13-b95912955e9d" (UID: "afe76cc1-30c2-4089-aa13-b95912955e9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:21:09.367810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367689 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-socket" pod "afe76cc1-30c2-4089-aa13-b95912955e9d" (UID: "afe76cc1-30c2-4089-aa13-b95912955e9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:21:09.367810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367746 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe76cc1-30c2-4089-aa13-b95912955e9d-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:09.367810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367765 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:09.367810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367779 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:09.367810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367788 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:09.367810 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.367796 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:09.369358 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.369324 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-envoy" pod "afe76cc1-30c2-4089-aa13-b95912955e9d" (UID: "afe76cc1-30c2-4089-aa13-b95912955e9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:21:09.369477 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.369352 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-podinfo" pod "afe76cc1-30c2-4089-aa13-b95912955e9d" (UID: "afe76cc1-30c2-4089-aa13-b95912955e9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:21:09.369477 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.369418 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-token" pod "afe76cc1-30c2-4089-aa13-b95912955e9d" (UID: "afe76cc1-30c2-4089-aa13-b95912955e9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:21:09.369587 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.369515 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-kube-api-access-kstdw" pod "afe76cc1-30c2-4089-aa13-b95912955e9d" (UID: "afe76cc1-30c2-4089-aa13-b95912955e9d"). InnerVolumeSpecName "kube-api-access-kstdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:21:09.469073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.469038 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:09.469073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.469066 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:09.469073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.469076 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:09.469365 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.469084 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kstdw\" (UniqueName: \"kubernetes.io/projected/afe76cc1-30c2-4089-aa13-b95912955e9d-kube-api-access-kstdw\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:21:09.497716 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.497683 4903 generic.go:356] "Generic (PLEG): container finished" podID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerID="7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710" exitCode=0 Nov 03 23:21:09.497896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.497756 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" Nov 03 23:21:09.497896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.497773 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" event={"ID":"afe76cc1-30c2-4089-aa13-b95912955e9d","Type":"ContainerDied","Data":"7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710"} Nov 03 23:21:09.497896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.497828 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2" event={"ID":"afe76cc1-30c2-4089-aa13-b95912955e9d","Type":"ContainerDied","Data":"7eda0dbae4e5b24465e5d9a977a74c7f4366bb01608434484feda014686fb38b"} Nov 03 23:21:09.497896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.497850 4903 scope.go:122] "RemoveContainer" containerID="7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710" Nov 03 23:21:09.507796 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.507777 4903 scope.go:122] "RemoveContainer" containerID="331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934" Nov 03 23:21:09.515746 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.515726 4903 scope.go:122] "RemoveContainer" containerID="7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710" Nov 03 23:21:09.516023 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:21:09.516006 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710\": container with ID starting with 7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710 not found: ID does not exist" containerID="7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710" Nov 03 23:21:09.516078 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.516029 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710"} err="failed to get container status \"7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710\": rpc error: code = NotFound desc = could not find container \"7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710\": container with ID starting with 7e4db706c50b8a489aae5052bfa8d119901d1fff1a5f25c34339c8b8a9f2e710 not found: ID does not exist" Nov 03 23:21:09.516078 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.516045 4903 scope.go:122] "RemoveContainer" containerID="331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934" Nov 03 23:21:09.516318 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:21:09.516288 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934\": container with ID starting with 331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934 not found: ID does not exist" containerID="331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934" Nov 03 23:21:09.516377 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.516324 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934"} err="failed to get container status \"331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934\": rpc error: code = NotFound desc = could not find container \"331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934\": container with ID starting with 331f93fc2aecd6603ae3c78963ac6df630ddb6ca77619362eb06cc69016eb934 not found: ID does not exist" Nov 03 23:21:09.545186 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.545159 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2"] Nov 03 23:21:09.553312 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.553275 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a6ecd9bf-openshift-default-8f5fcfb57-8b7z2"] Nov 03 23:21:09.731644 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:09.731614 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" path="/var/lib/kubelet/pods/afe76cc1-30c2-4089-aa13-b95912955e9d/volumes" Nov 03 23:21:10.645468 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.645429 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j"] Nov 03 23:21:10.645871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.645807 4903 cpu_manager.go:414] "containerMap: removing container" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" Nov 03 23:21:10.645871 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.645818 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" Nov 03 23:21:10.645942 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.645902 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" Nov 03 23:21:10.645942 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.645911 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" Nov 03 23:21:10.649266 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.649248 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:10.675235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.675207 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-02e08d-dockercfg-928zg\"" Nov 03 23:21:10.675235 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.675221 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-02e08d\"" Nov 03 23:21:10.676992 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.676973 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j"] Nov 03 23:21:10.711520 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.711486 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6"] Nov 03 23:21:10.711958 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.711941 4903 cpu_manager.go:414] "containerMap: removing container" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" Nov 03 23:21:10.712070 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.711961 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe76cc1-30c2-4089-aa13-b95912955e9d" containerName="istio-proxy" Nov 03 23:21:10.714215 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.714194 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:10.733163 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.733137 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6"] Nov 03 23:21:10.782367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.782330 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334ca29b-728f-40b8-bd19-ee76d4b274f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6\" (UID: \"334ca29b-728f-40b8-bd19-ee76d4b274f7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:10.782596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.782390 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jqq\" (UniqueName: \"kubernetes.io/projected/c33324f7-f71b-4697-8e65-23608ee505ef-kube-api-access-v9jqq\") pod \"payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j\" (UID: \"c33324f7-f71b-4697-8e65-23608ee505ef\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:10.782596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.782467 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv296\" (UniqueName: \"kubernetes.io/projected/334ca29b-728f-40b8-bd19-ee76d4b274f7-kube-api-access-jv296\") pod \"payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6\" (UID: \"334ca29b-728f-40b8-bd19-ee76d4b274f7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:10.782596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.782512 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33324f7-f71b-4697-8e65-23608ee505ef-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j\" (UID: \"c33324f7-f71b-4697-8e65-23608ee505ef\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:10.883380 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.883341 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334ca29b-728f-40b8-bd19-ee76d4b274f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6\" (UID: \"334ca29b-728f-40b8-bd19-ee76d4b274f7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:10.883576 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.883388 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jqq\" (UniqueName: \"kubernetes.io/projected/c33324f7-f71b-4697-8e65-23608ee505ef-kube-api-access-v9jqq\") pod \"payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j\" (UID: \"c33324f7-f71b-4697-8e65-23608ee505ef\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:10.883576 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.883451 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv296\" (UniqueName: \"kubernetes.io/projected/334ca29b-728f-40b8-bd19-ee76d4b274f7-kube-api-access-jv296\") pod \"payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6\" (UID: \"334ca29b-728f-40b8-bd19-ee76d4b274f7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:10.883681 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.883577 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33324f7-f71b-4697-8e65-23608ee505ef-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j\" (UID: \"c33324f7-f71b-4697-8e65-23608ee505ef\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:10.884171 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.884143 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334ca29b-728f-40b8-bd19-ee76d4b274f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6\" (UID: \"334ca29b-728f-40b8-bd19-ee76d4b274f7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:10.884349 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.884327 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33324f7-f71b-4697-8e65-23608ee505ef-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j\" (UID: \"c33324f7-f71b-4697-8e65-23608ee505ef\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:10.949691 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.949660 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv296\" (UniqueName: \"kubernetes.io/projected/334ca29b-728f-40b8-bd19-ee76d4b274f7-kube-api-access-jv296\") pod \"payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6\" (UID: \"334ca29b-728f-40b8-bd19-ee76d4b274f7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:10.957267 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.957241 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jqq\" (UniqueName: \"kubernetes.io/projected/c33324f7-f71b-4697-8e65-23608ee505ef-kube-api-access-v9jqq\") pod \"payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j\" (UID: \"c33324f7-f71b-4697-8e65-23608ee505ef\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:10.961992 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:10.961973 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:11.026707 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.026672 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:11.258305 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.258267 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j"] Nov 03 23:21:11.260525 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:21:11.260501 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33324f7_f71b_4697_8e65_23608ee505ef.slice/crio-18d082db1fba0135b217b5e63bce1f39aeeebc1becf413dca60a6a4c29dc39ea WatchSource:0}: Error finding container 18d082db1fba0135b217b5e63bce1f39aeeebc1becf413dca60a6a4c29dc39ea: Status 404 returned error can't find the container with id 18d082db1fba0135b217b5e63bce1f39aeeebc1becf413dca60a6a4c29dc39ea Nov 03 23:21:11.325394 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.325366 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6"] Nov 03 23:21:11.328191 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:21:11.328166 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334ca29b_728f_40b8_bd19_ee76d4b274f7.slice/crio-f34b4fd6cd18d27c377965054d850bab696aab3aaed7b3bcc8251a721cd6eec8 WatchSource:0}: Error finding container f34b4fd6cd18d27c377965054d850bab696aab3aaed7b3bcc8251a721cd6eec8: Status 404 returned error can't find the container with id f34b4fd6cd18d27c377965054d850bab696aab3aaed7b3bcc8251a721cd6eec8 Nov 03 23:21:11.512094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.511995 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" event={"ID":"334ca29b-728f-40b8-bd19-ee76d4b274f7","Type":"ContainerStarted","Data":"519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd"} Nov 03 23:21:11.512094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.512060 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" event={"ID":"334ca29b-728f-40b8-bd19-ee76d4b274f7","Type":"ContainerStarted","Data":"f34b4fd6cd18d27c377965054d850bab696aab3aaed7b3bcc8251a721cd6eec8"} Nov 03 23:21:11.512371 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.512098 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:11.513601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.513577 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" event={"ID":"c33324f7-f71b-4697-8e65-23608ee505ef","Type":"ContainerStarted","Data":"a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214"} Nov 03 23:21:11.513711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.513616 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" event={"ID":"c33324f7-f71b-4697-8e65-23608ee505ef","Type":"ContainerStarted","Data":"18d082db1fba0135b217b5e63bce1f39aeeebc1becf413dca60a6a4c29dc39ea"} Nov 03 23:21:11.513711 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.513668 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:11.552064 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.552018 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" podStartSLOduration=1.552004519 podStartE2EDuration="1.552004519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:21:11.550436289 +0000 UTC m=+4097.954377264" watchObservedRunningTime="2026-11-03 23:21:11.552004519 +0000 UTC m=+4097.955945506" Nov 03 23:21:11.587097 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:11.587045 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" podStartSLOduration=1.587030623 podStartE2EDuration="1.587030623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:21:11.586371903 +0000 UTC m=+4097.990312878" watchObservedRunningTime="2026-11-03 23:21:11.587030623 +0000 UTC m=+4097.990971597" Nov 03 23:21:14.702815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:14.702773 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:21:18.389871 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:21:18.389830 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:21:22.521766 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:22.521735 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:21:22.522332 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:22.521797 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:21:23.528439 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:23.528391 4903 patch_prober.go:29] interesting pod/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 23:21:23.528965 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:23.528453 4903 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 23:21:24.614463 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:24.614380 4903 generic.go:356] "Generic (PLEG): container finished" podID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerID="91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0" exitCode=0 Nov 03 23:21:24.614896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:24.614454 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" event={"ID":"05b97e1d-aa29-4d50-ac25-2236b01aab43","Type":"ContainerDied","Data":"91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0"} Nov 03 23:21:24.614896 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:24.614829 4903 scope.go:122] "RemoveContainer" containerID="91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0" Nov 03 23:21:24.616083 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:24.616058 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:21:24.616160 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:24.616124 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:21:24.616160 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:24.616155 4903 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 23:21:25.625207 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:25.625162 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" event={"ID":"05b97e1d-aa29-4d50-ac25-2236b01aab43","Type":"ContainerStarted","Data":"91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd"} Nov 03 23:21:26.058773 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:26.058736 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:26.059168 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:26.059138 4903 patch_prober.go:29] interesting pod/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Nov 03 23:21:26.059399 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:26.059371 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Nov 03 23:21:27.065454 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:27.065414 4903 patch_prober.go:29] interesting pod/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Nov 03 23:21:27.065890 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:27.065473 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Nov 03 23:21:28.061441 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:28.061399 4903 patch_prober.go:29] interesting pod/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Nov 03 23:21:28.061756 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:28.061724 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Nov 03 23:21:29.062315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:29.061705 4903 patch_prober.go:29] interesting pod/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Nov 03 23:21:29.062315 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:29.061768 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Nov 03 23:21:30.063318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:30.059738 4903 patch_prober.go:29] interesting pod/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" start-of-body= Nov 03 23:21:30.063318 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:30.059817 4903 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.158:15021/healthz/ready\": dial tcp 10.132.1.158:15021: connect: connection refused" Nov 03 23:21:31.058831 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:31.058796 4903 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:31.059148 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:31.059127 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:31.059839 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:21:31.059819 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:21:48.383588 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:21:48.383549 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:22:01.834549 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:01.834492 4903 scope.go:122] "RemoveContainer" containerID="7af06d5e5314ed249af1ab4ff787d645f3fa54b1bb64cf68604fd0b3cf5f4a65" Nov 03 23:22:18.383414 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:22:18.383371 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:22:43.908121 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:43.908040 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd"] Nov 03 23:22:43.911427 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:43.911405 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:43.924158 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:43.924141 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:22:43.924266 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:43.924205 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:22:43.934501 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:43.934474 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:22:43.934501 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:43.934476 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gjww\"" Nov 03 23:22:43.936226 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:43.936205 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd"] Nov 03 23:22:44.012574 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.012537 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.012574 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.012583 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5cc4\" (UniqueName: \"kubernetes.io/projected/5e59da96-4f2b-4651-853f-c335bac9fe20-kube-api-access-l5cc4\") pod \"maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.012815 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.012698 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.114251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.114200 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.114251 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.114257 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5cc4\" (UniqueName: \"kubernetes.io/projected/5e59da96-4f2b-4651-853f-c335bac9fe20-kube-api-access-l5cc4\") pod \"maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.114497 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.114317 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.114956 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.114932 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.116596 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.116567 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.135577 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.135552 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5cc4\" (UniqueName: \"kubernetes.io/projected/5e59da96-4f2b-4651-853f-c335bac9fe20-kube-api-access-l5cc4\") pod \"maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.224461 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.224385 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:44.526383 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:44.526358 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd"] Nov 03 23:22:44.528946 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:22:44.528918 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e59da96_4f2b_4651_853f_c335bac9fe20.slice/crio-3c9e0bc7fd0dead759b0193325969bea53ff50833ff274817d50a92fb2199ad1 WatchSource:0}: Error finding container 3c9e0bc7fd0dead759b0193325969bea53ff50833ff274817d50a92fb2199ad1: Status 404 returned error can't find the container with id 3c9e0bc7fd0dead759b0193325969bea53ff50833ff274817d50a92fb2199ad1 Nov 03 23:22:45.029035 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:45.029000 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" event={"ID":"5e59da96-4f2b-4651-853f-c335bac9fe20","Type":"ContainerStarted","Data":"c4132a7996914b9cd608195f3f584702e10bd94d37308bb9fc2ee0982032682b"} Nov 03 23:22:45.029582 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:45.029049 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" event={"ID":"5e59da96-4f2b-4651-853f-c335bac9fe20","Type":"ContainerStarted","Data":"3c9e0bc7fd0dead759b0193325969bea53ff50833ff274817d50a92fb2199ad1"} Nov 03 23:22:46.034980 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:46.034942 4903 generic.go:356] "Generic (PLEG): container finished" podID="5e59da96-4f2b-4651-853f-c335bac9fe20" containerID="c4132a7996914b9cd608195f3f584702e10bd94d37308bb9fc2ee0982032682b" exitCode=0 Nov 03 23:22:46.035439 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:46.035028 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" event={"ID":"5e59da96-4f2b-4651-853f-c335bac9fe20","Type":"ContainerDied","Data":"c4132a7996914b9cd608195f3f584702e10bd94d37308bb9fc2ee0982032682b"} Nov 03 23:22:47.178287 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.178260 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:47.344188 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.344085 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-openshift-ca\") pod \"5e59da96-4f2b-4651-853f-c335bac9fe20\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " Nov 03 23:22:47.344188 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.344141 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-xks-ca\") pod \"5e59da96-4f2b-4651-853f-c335bac9fe20\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " Nov 03 23:22:47.344471 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.344197 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e59da96-4f2b-4651-853f-c335bac9fe20-kube-api-access-l5cc4\" (UniqueName: \"kubernetes.io/projected/5e59da96-4f2b-4651-853f-c335bac9fe20-kube-api-access-l5cc4\") pod \"5e59da96-4f2b-4651-853f-c335bac9fe20\" (UID: \"5e59da96-4f2b-4651-853f-c335bac9fe20\") " Nov 03 23:22:47.344608 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.344567 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-openshift-ca" pod "5e59da96-4f2b-4651-853f-c335bac9fe20" (UID: "5e59da96-4f2b-4651-853f-c335bac9fe20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:47.346365 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.346337 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-xks-ca" pod "5e59da96-4f2b-4651-853f-c335bac9fe20" (UID: "5e59da96-4f2b-4651-853f-c335bac9fe20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:22:47.346471 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.346412 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e59da96-4f2b-4651-853f-c335bac9fe20-kube-api-access-l5cc4" pod "5e59da96-4f2b-4651-853f-c335bac9fe20" (UID: "5e59da96-4f2b-4651-853f-c335bac9fe20"). InnerVolumeSpecName "kube-api-access-l5cc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:47.446026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.445986 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5cc4\" (UniqueName: \"kubernetes.io/projected/5e59da96-4f2b-4651-853f-c335bac9fe20-kube-api-access-l5cc4\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:22:47.446026 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.446022 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:22:47.446249 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:47.446037 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59da96-4f2b-4651-853f-c335bac9fe20-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:22:48.046945 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:48.046861 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" Nov 03 23:22:48.046945 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:48.046870 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd" event={"ID":"5e59da96-4f2b-4651-853f-c335bac9fe20","Type":"ContainerDied","Data":"3c9e0bc7fd0dead759b0193325969bea53ff50833ff274817d50a92fb2199ad1"} Nov 03 23:22:48.046945 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:48.046918 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c9e0bc7fd0dead759b0193325969bea53ff50833ff274817d50a92fb2199ad1" Nov 03 23:22:48.383620 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:22:48.383534 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:22:54.321707 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:54.321624 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j"] Nov 03 23:22:54.350499 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:54.350465 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6"] Nov 03 23:22:54.525352 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:54.525324 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:22:54.532861 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:54.532836 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:22:54.534813 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:54.534788 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:22:54.543162 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:54.543144 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djcf8_44c8b7d7-71ad-47ee-9aad-be7f366974f5/ovn-acl-logging/0.log" Nov 03 23:22:59.322833 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.322791 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" podUID="c33324f7-f71b-4697-8e65-23608ee505ef" containerName="payload-processing" containerID="cri-o://a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214" gracePeriod=25 Nov 03 23:22:59.351658 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.351590 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" podUID="334ca29b-728f-40b8-bd19-ee76d4b274f7" containerName="payload-pre-processing" containerID="cri-o://519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd" gracePeriod=25 Nov 03 23:22:59.616036 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.616005 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:22:59.643865 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.643841 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:22:59.665171 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.665142 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334ca29b-728f-40b8-bd19-ee76d4b274f7-kube-api-access-jv296\" (UniqueName: \"kubernetes.io/projected/334ca29b-728f-40b8-bd19-ee76d4b274f7-kube-api-access-jv296\") pod \"334ca29b-728f-40b8-bd19-ee76d4b274f7\" (UID: \"334ca29b-728f-40b8-bd19-ee76d4b274f7\") " Nov 03 23:22:59.665338 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.665268 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33324f7-f71b-4697-8e65-23608ee505ef-kube-api-access-v9jqq\" (UniqueName: \"kubernetes.io/projected/c33324f7-f71b-4697-8e65-23608ee505ef-kube-api-access-v9jqq\") pod \"c33324f7-f71b-4697-8e65-23608ee505ef\" (UID: \"c33324f7-f71b-4697-8e65-23608ee505ef\") " Nov 03 23:22:59.665338 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.665332 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33324f7-f71b-4697-8e65-23608ee505ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33324f7-f71b-4697-8e65-23608ee505ef-plugins-config-volume\") pod \"c33324f7-f71b-4697-8e65-23608ee505ef\" (UID: \"c33324f7-f71b-4697-8e65-23608ee505ef\") " Nov 03 23:22:59.665490 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.665366 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334ca29b-728f-40b8-bd19-ee76d4b274f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334ca29b-728f-40b8-bd19-ee76d4b274f7-plugins-config-volume\") pod \"334ca29b-728f-40b8-bd19-ee76d4b274f7\" (UID: \"334ca29b-728f-40b8-bd19-ee76d4b274f7\") " Nov 03 23:22:59.665937 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.665909 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334ca29b-728f-40b8-bd19-ee76d4b274f7-plugins-config-volume" pod "334ca29b-728f-40b8-bd19-ee76d4b274f7" (UID: "334ca29b-728f-40b8-bd19-ee76d4b274f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:59.666051 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.665907 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33324f7-f71b-4697-8e65-23608ee505ef-plugins-config-volume" pod "c33324f7-f71b-4697-8e65-23608ee505ef" (UID: "c33324f7-f71b-4697-8e65-23608ee505ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:22:59.667731 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.667704 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33324f7-f71b-4697-8e65-23608ee505ef-kube-api-access-v9jqq" pod "c33324f7-f71b-4697-8e65-23608ee505ef" (UID: "c33324f7-f71b-4697-8e65-23608ee505ef"). InnerVolumeSpecName "kube-api-access-v9jqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:59.668052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.668021 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334ca29b-728f-40b8-bd19-ee76d4b274f7-kube-api-access-jv296" pod "334ca29b-728f-40b8-bd19-ee76d4b274f7" (UID: "334ca29b-728f-40b8-bd19-ee76d4b274f7"). InnerVolumeSpecName "kube-api-access-jv296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:22:59.766122 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.766091 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv296\" (UniqueName: \"kubernetes.io/projected/334ca29b-728f-40b8-bd19-ee76d4b274f7-kube-api-access-jv296\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:22:59.766122 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.766123 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9jqq\" (UniqueName: \"kubernetes.io/projected/c33324f7-f71b-4697-8e65-23608ee505ef-kube-api-access-v9jqq\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:22:59.766392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.766136 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c33324f7-f71b-4697-8e65-23608ee505ef-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:22:59.766392 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:22:59.766145 4903 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334ca29b-728f-40b8-bd19-ee76d4b274f7-plugins-config-volume\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:00.108452 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.108414 4903 generic.go:356] "Generic (PLEG): container finished" podID="334ca29b-728f-40b8-bd19-ee76d4b274f7" containerID="519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd" exitCode=0 Nov 03 23:23:00.108663 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.108480 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" Nov 03 23:23:00.108663 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.108497 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" event={"ID":"334ca29b-728f-40b8-bd19-ee76d4b274f7","Type":"ContainerDied","Data":"519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd"} Nov 03 23:23:00.108663 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.108587 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6" event={"ID":"334ca29b-728f-40b8-bd19-ee76d4b274f7","Type":"ContainerDied","Data":"f34b4fd6cd18d27c377965054d850bab696aab3aaed7b3bcc8251a721cd6eec8"} Nov 03 23:23:00.108663 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.108621 4903 scope.go:122] "RemoveContainer" containerID="519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd" Nov 03 23:23:00.109853 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.109836 4903 generic.go:356] "Generic (PLEG): container finished" podID="c33324f7-f71b-4697-8e65-23608ee505ef" containerID="a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214" exitCode=0 Nov 03 23:23:00.109960 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.109902 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" Nov 03 23:23:00.109960 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.109926 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" event={"ID":"c33324f7-f71b-4697-8e65-23608ee505ef","Type":"ContainerDied","Data":"a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214"} Nov 03 23:23:00.110076 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.109971 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j" event={"ID":"c33324f7-f71b-4697-8e65-23608ee505ef","Type":"ContainerDied","Data":"18d082db1fba0135b217b5e63bce1f39aeeebc1becf413dca60a6a4c29dc39ea"} Nov 03 23:23:00.117762 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.117747 4903 scope.go:122] "RemoveContainer" containerID="519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd" Nov 03 23:23:00.118025 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:23:00.118005 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd\": container with ID starting with 519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd not found: ID does not exist" containerID="519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd" Nov 03 23:23:00.118073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.118031 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd"} err="failed to get container status \"519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd\": rpc error: code = NotFound desc = could not find container \"519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd\": container with ID starting with 519087bad123505aaeb9b083a7b8299bed3b969b4ac646990dbe6da5bdbb6ffd not found: ID does not exist" Nov 03 23:23:00.118073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.118047 4903 scope.go:122] "RemoveContainer" containerID="a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214" Nov 03 23:23:00.126034 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.126013 4903 scope.go:122] "RemoveContainer" containerID="a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214" Nov 03 23:23:00.126307 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:23:00.126268 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214\": container with ID starting with a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214 not found: ID does not exist" containerID="a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214" Nov 03 23:23:00.126387 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.126316 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214"} err="failed to get container status \"a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214\": rpc error: code = NotFound desc = could not find container \"a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214\": container with ID starting with a1ff4b341881508380f25c546e26473f09a7107ee742b79d3726096fa5cb6214 not found: ID does not exist" Nov 03 23:23:00.153738 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.153705 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j"] Nov 03 23:23:00.163772 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.163739 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-02e08d-7d74877466-xrf9j"] Nov 03 23:23:00.188547 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.188516 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6"] Nov 03 23:23:00.192718 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:00.192694 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-02e08d-6f4945bb79-sxdq6"] Nov 03 23:23:01.733266 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:01.733224 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334ca29b-728f-40b8-bd19-ee76d4b274f7" path="/var/lib/kubelet/pods/334ca29b-728f-40b8-bd19-ee76d4b274f7/volumes" Nov 03 23:23:01.733776 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:01.733760 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33324f7-f71b-4697-8e65-23608ee505ef" path="/var/lib/kubelet/pods/c33324f7-f71b-4697-8e65-23608ee505ef/volumes" Nov 03 23:23:02.149685 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:02.149599 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd"] Nov 03 23:23:02.159568 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:02.159537 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-02e08d-vwjpd"] Nov 03 23:23:03.731634 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:03.731588 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e59da96-4f2b-4651-853f-c335bac9fe20" path="/var/lib/kubelet/pods/5e59da96-4f2b-4651-853f-c335bac9fe20/volumes" Nov 03 23:23:04.614959 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:04.614925 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:23:09.214739 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:09.214693 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz"] Nov 03 23:23:09.215371 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:09.214970 4903 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" containerID="cri-o://91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd" gracePeriod=30 Nov 03 23:23:11.906097 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:11.906059 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:23:14.741502 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.741471 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:23:14.826797 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.826702 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-socket\") pod \"05b97e1d-aa29-4d50-ac25-2236b01aab43\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " Nov 03 23:23:14.826797 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.826767 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-podinfo\") pod \"05b97e1d-aa29-4d50-ac25-2236b01aab43\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " Nov 03 23:23:14.827231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.826829 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b97e1d-aa29-4d50-ac25-2236b01aab43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b97e1d-aa29-4d50-ac25-2236b01aab43-istiod-ca-cert\") pod \"05b97e1d-aa29-4d50-ac25-2236b01aab43\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " Nov 03 23:23:14.827231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.826866 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-certs\") pod \"05b97e1d-aa29-4d50-ac25-2236b01aab43\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " Nov 03 23:23:14.827231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.826939 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-credential-socket\") pod \"05b97e1d-aa29-4d50-ac25-2236b01aab43\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " Nov 03 23:23:14.827231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.827000 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-kube-api-access-dbbrp\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-kube-api-access-dbbrp\") pod \"05b97e1d-aa29-4d50-ac25-2236b01aab43\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " Nov 03 23:23:14.827231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.827043 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-token\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-token\") pod \"05b97e1d-aa29-4d50-ac25-2236b01aab43\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " Nov 03 23:23:14.827231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.827069 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-envoy\") pod \"05b97e1d-aa29-4d50-ac25-2236b01aab43\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " Nov 03 23:23:14.827231 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.827109 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-data\") pod \"05b97e1d-aa29-4d50-ac25-2236b01aab43\" (UID: \"05b97e1d-aa29-4d50-ac25-2236b01aab43\") " Nov 03 23:23:14.827743 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.827455 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-certs" pod "05b97e1d-aa29-4d50-ac25-2236b01aab43" (UID: "05b97e1d-aa29-4d50-ac25-2236b01aab43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:14.827838 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.827813 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-credential-socket" pod "05b97e1d-aa29-4d50-ac25-2236b01aab43" (UID: "05b97e1d-aa29-4d50-ac25-2236b01aab43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:14.827962 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.827936 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-data" pod "05b97e1d-aa29-4d50-ac25-2236b01aab43" (UID: "05b97e1d-aa29-4d50-ac25-2236b01aab43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:14.828117 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.828060 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b97e1d-aa29-4d50-ac25-2236b01aab43-istiod-ca-cert" pod "05b97e1d-aa29-4d50-ac25-2236b01aab43" (UID: "05b97e1d-aa29-4d50-ac25-2236b01aab43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:23:14.829240 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.829217 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-socket" pod "05b97e1d-aa29-4d50-ac25-2236b01aab43" (UID: "05b97e1d-aa29-4d50-ac25-2236b01aab43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:14.830799 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.830772 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-token" pod "05b97e1d-aa29-4d50-ac25-2236b01aab43" (UID: "05b97e1d-aa29-4d50-ac25-2236b01aab43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:23:14.841781 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.841755 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-kube-api-access-dbbrp" pod "05b97e1d-aa29-4d50-ac25-2236b01aab43" (UID: "05b97e1d-aa29-4d50-ac25-2236b01aab43"). InnerVolumeSpecName "kube-api-access-dbbrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:23:14.842503 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.842479 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-podinfo" pod "05b97e1d-aa29-4d50-ac25-2236b01aab43" (UID: "05b97e1d-aa29-4d50-ac25-2236b01aab43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 23:23:14.844775 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.844748 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-envoy" pod "05b97e1d-aa29-4d50-ac25-2236b01aab43" (UID: "05b97e1d-aa29-4d50-ac25-2236b01aab43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 23:23:14.928924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.928890 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbrp\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-kube-api-access-dbbrp\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:14.928924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.928921 4903 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-token\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:14.928924 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.928931 4903 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-envoy\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:14.929173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.928939 4903 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-data\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:14.929173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.928947 4903 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:14.929173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.928957 4903 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05b97e1d-aa29-4d50-ac25-2236b01aab43-istio-podinfo\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:14.929173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.928965 4903 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05b97e1d-aa29-4d50-ac25-2236b01aab43-istiod-ca-cert\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:14.929173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.928973 4903 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-workload-certs\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:14.929173 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:14.928982 4903 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05b97e1d-aa29-4d50-ac25-2236b01aab43-credential-socket\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:15.206161 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.206118 4903 generic.go:356] "Generic (PLEG): container finished" podID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerID="91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd" exitCode=0 Nov 03 23:23:15.206402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.206173 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" event={"ID":"05b97e1d-aa29-4d50-ac25-2236b01aab43","Type":"ContainerDied","Data":"91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd"} Nov 03 23:23:15.206402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.206232 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" event={"ID":"05b97e1d-aa29-4d50-ac25-2236b01aab43","Type":"ContainerDied","Data":"4277baba31b3ae9c232b8296fdf2b99ea96bd071dcd2af83a4574eb46089c298"} Nov 03 23:23:15.206402 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.206282 4903 scope.go:122] "RemoveContainer" containerID="91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd" Nov 03 23:23:15.206604 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.206483 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz" Nov 03 23:23:15.217489 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.217467 4903 scope.go:122] "RemoveContainer" containerID="91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0" Nov 03 23:23:15.226188 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.226166 4903 scope.go:122] "RemoveContainer" containerID="91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd" Nov 03 23:23:15.226483 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:23:15.226463 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd\": container with ID starting with 91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd not found: ID does not exist" containerID="91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd" Nov 03 23:23:15.226556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.226491 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd"} err="failed to get container status \"91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd\": rpc error: code = NotFound desc = could not find container \"91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd\": container with ID starting with 91ffe2f4856e8950955c1754c541bb7d25ba349af4b9a8b67fc967f2f6df96cd not found: ID does not exist" Nov 03 23:23:15.226556 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.226509 4903 scope.go:122] "RemoveContainer" containerID="91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0" Nov 03 23:23:15.226764 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:23:15.226746 4903 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0\": container with ID starting with 91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0 not found: ID does not exist" containerID="91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0" Nov 03 23:23:15.226807 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.226771 4903 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0"} err="failed to get container status \"91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0\": rpc error: code = NotFound desc = could not find container \"91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0\": container with ID starting with 91b94f0fb06ce073b32910c6b8373468cb89946750fff34e717b39c556cf60b0 not found: ID does not exist" Nov 03 23:23:15.252221 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.252193 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz"] Nov 03 23:23:15.262613 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.262581 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-02e08d-openshift-default-8d4777cb5-mdkrz"] Nov 03 23:23:15.731111 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:15.731076 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" path="/var/lib/kubelet/pods/05b97e1d-aa29-4d50-ac25-2236b01aab43/volumes" Nov 03 23:23:18.382843 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:23:18.382794 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:23:20.023073 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:20.023033 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:23:21.027545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:21.027502 4903 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq"] Nov 03 23:23:21.037855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:21.037815 4903 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b617c1d-s5bfq"] Nov 03 23:23:21.733001 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:21.732960 4903 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc640593-e132-47fa-adfb-f3d02ad55962" path="/var/lib/kubelet/pods/cc640593-e132-47fa-adfb-f3d02ad55962/volumes" Nov 03 23:23:26.708142 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708107 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb"] Nov 03 23:23:26.708611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708517 4903 cpu_manager.go:414] "containerMap: removing container" podUID="c33324f7-f71b-4697-8e65-23608ee505ef" containerName="payload-processing" Nov 03 23:23:26.708611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708534 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33324f7-f71b-4697-8e65-23608ee505ef" containerName="payload-processing" Nov 03 23:23:26.708611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708552 4903 cpu_manager.go:414] "containerMap: removing container" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" Nov 03 23:23:26.708611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708560 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" Nov 03 23:23:26.708611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708575 4903 cpu_manager.go:414] "containerMap: removing container" podUID="5e59da96-4f2b-4651-853f-c335bac9fe20" containerName="revoke-keys" Nov 03 23:23:26.708611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708583 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e59da96-4f2b-4651-853f-c335bac9fe20" containerName="revoke-keys" Nov 03 23:23:26.708611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708605 4903 cpu_manager.go:414] "containerMap: removing container" podUID="334ca29b-728f-40b8-bd19-ee76d4b274f7" containerName="payload-pre-processing" Nov 03 23:23:26.708611 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708612 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334ca29b-728f-40b8-bd19-ee76d4b274f7" containerName="payload-pre-processing" Nov 03 23:23:26.709119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708704 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" Nov 03 23:23:26.709119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708718 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33324f7-f71b-4697-8e65-23608ee505ef" containerName="payload-processing" Nov 03 23:23:26.709119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708730 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e59da96-4f2b-4651-853f-c335bac9fe20" containerName="revoke-keys" Nov 03 23:23:26.709119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.708743 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="334ca29b-728f-40b8-bd19-ee76d4b274f7" containerName="payload-pre-processing" Nov 03 23:23:26.713645 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.713627 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:26.731198 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.731176 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb"] Nov 03 23:23:26.736771 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.736749 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:23:26.736891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.736768 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:23:26.736891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.736776 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gjww\"" Nov 03 23:23:26.736891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.736843 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:23:26.757014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.756983 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lgbpb\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:26.757014 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.757016 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8lg\" (UniqueName: \"kubernetes.io/projected/84ccae5c-0781-4ddb-9422-8d8dcae660cc-kube-api-access-ws8lg\") pod \"maas-api-revoke-sub-f03629db-lgbpb\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:26.757220 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.757199 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lgbpb\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:26.858706 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.858669 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lgbpb\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:26.858891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.858737 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lgbpb\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:26.858891 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.858757 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8lg\" (UniqueName: \"kubernetes.io/projected/84ccae5c-0781-4ddb-9422-8d8dcae660cc-kube-api-access-ws8lg\") pod \"maas-api-revoke-sub-f03629db-lgbpb\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:26.859367 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.859336 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lgbpb\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:26.861056 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.861030 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lgbpb\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:26.880598 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:26.880573 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8lg\" (UniqueName: \"kubernetes.io/projected/84ccae5c-0781-4ddb-9422-8d8dcae660cc-kube-api-access-ws8lg\") pod \"maas-api-revoke-sub-f03629db-lgbpb\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:27.027951 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:27.027849 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:27.326420 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:27.326368 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb"] Nov 03 23:23:27.328706 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:23:27.328671 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ccae5c_0781_4ddb_9422_8d8dcae660cc.slice/crio-d4f09f8ce776b456fffae333739754971c3b85f66cf01d620fb6211a0145f3a3 WatchSource:0}: Error finding container d4f09f8ce776b456fffae333739754971c3b85f66cf01d620fb6211a0145f3a3: Status 404 returned error can't find the container with id d4f09f8ce776b456fffae333739754971c3b85f66cf01d620fb6211a0145f3a3 Nov 03 23:23:28.301397 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:28.301365 4903 generic.go:356] "Generic (PLEG): container finished" podID="84ccae5c-0781-4ddb-9422-8d8dcae660cc" containerID="3f589afae46bfc95a95b7028dd0604dd26ce50ee8a087b531f3bdca0b73f959b" exitCode=0 Nov 03 23:23:28.301797 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:28.301445 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" event={"ID":"84ccae5c-0781-4ddb-9422-8d8dcae660cc","Type":"ContainerDied","Data":"3f589afae46bfc95a95b7028dd0604dd26ce50ee8a087b531f3bdca0b73f959b"} Nov 03 23:23:28.301797 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:28.301496 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" event={"ID":"84ccae5c-0781-4ddb-9422-8d8dcae660cc","Type":"ContainerStarted","Data":"d4f09f8ce776b456fffae333739754971c3b85f66cf01d620fb6211a0145f3a3"} Nov 03 23:23:29.440601 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.440576 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:29.480585 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.480544 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ccae5c-0781-4ddb-9422-8d8dcae660cc-kube-api-access-ws8lg\" (UniqueName: \"kubernetes.io/projected/84ccae5c-0781-4ddb-9422-8d8dcae660cc-kube-api-access-ws8lg\") pod \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " Nov 03 23:23:29.480740 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.480606 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-xks-ca\") pod \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " Nov 03 23:23:29.480740 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.480704 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-openshift-ca\") pod \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\" (UID: \"84ccae5c-0781-4ddb-9422-8d8dcae660cc\") " Nov 03 23:23:29.481119 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.481090 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-openshift-ca" pod "84ccae5c-0781-4ddb-9422-8d8dcae660cc" (UID: "84ccae5c-0781-4ddb-9422-8d8dcae660cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:23:29.482955 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.482928 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ccae5c-0781-4ddb-9422-8d8dcae660cc-kube-api-access-ws8lg" pod "84ccae5c-0781-4ddb-9422-8d8dcae660cc" (UID: "84ccae5c-0781-4ddb-9422-8d8dcae660cc"). InnerVolumeSpecName "kube-api-access-ws8lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:23:29.483097 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.482988 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-xks-ca" pod "84ccae5c-0781-4ddb-9422-8d8dcae660cc" (UID: "84ccae5c-0781-4ddb-9422-8d8dcae660cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:23:29.581916 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.581829 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:29.581916 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.581863 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84ccae5c-0781-4ddb-9422-8d8dcae660cc-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:29.581916 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:29.581872 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8lg\" (UniqueName: \"kubernetes.io/projected/84ccae5c-0781-4ddb-9422-8d8dcae660cc-kube-api-access-ws8lg\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:30.312483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:30.312453 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" Nov 03 23:23:30.312483 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:30.312466 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lgbpb" event={"ID":"84ccae5c-0781-4ddb-9422-8d8dcae660cc","Type":"ContainerDied","Data":"d4f09f8ce776b456fffae333739754971c3b85f66cf01d620fb6211a0145f3a3"} Nov 03 23:23:30.312704 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:30.312513 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4f09f8ce776b456fffae333739754971c3b85f66cf01d620fb6211a0145f3a3" Nov 03 23:23:37.995041 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:37.995004 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:23:43.663046 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.663012 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn"] Nov 03 23:23:43.663474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.663416 4903 cpu_manager.go:414] "containerMap: removing container" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" Nov 03 23:23:43.663474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.663435 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" Nov 03 23:23:43.663474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.663460 4903 cpu_manager.go:414] "containerMap: removing container" podUID="84ccae5c-0781-4ddb-9422-8d8dcae660cc" containerName="revoke-keys" Nov 03 23:23:43.663474 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.663465 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ccae5c-0781-4ddb-9422-8d8dcae660cc" containerName="revoke-keys" Nov 03 23:23:43.663628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.663528 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ccae5c-0781-4ddb-9422-8d8dcae660cc" containerName="revoke-keys" Nov 03 23:23:43.663628 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.663540 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b97e1d-aa29-4d50-ac25-2236b01aab43" containerName="istio-proxy" Nov 03 23:23:43.666537 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.666518 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.680157 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.680138 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:23:43.680284 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.680220 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:23:43.690734 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.690705 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:23:43.690940 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.690735 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gjww\"" Nov 03 23:23:43.692725 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.692701 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn"] Nov 03 23:23:43.727461 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.727420 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvntn\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.727673 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.727536 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvntn\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.727734 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.727669 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76k9s\" (UniqueName: \"kubernetes.io/projected/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-kube-api-access-76k9s\") pod \"maas-api-revoke-sub-0165d7d2-rvntn\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.828745 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.828703 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76k9s\" (UniqueName: \"kubernetes.io/projected/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-kube-api-access-76k9s\") pod \"maas-api-revoke-sub-0165d7d2-rvntn\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.828961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.828763 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvntn\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.828961 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.828803 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvntn\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.829490 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.829466 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvntn\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.831123 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.831104 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rvntn\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.850034 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.850011 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76k9s\" (UniqueName: \"kubernetes.io/projected/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-kube-api-access-76k9s\") pod \"maas-api-revoke-sub-0165d7d2-rvntn\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:43.979616 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:43.979545 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:44.277276 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:44.277247 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn"] Nov 03 23:23:44.280343 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:23:44.280308 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ee8fc7_1234_4e5b_a1e0_80420c88c0ff.slice/crio-77add0aaa0a1dbfe6beaf093e38eab56a0a7ad669b0d1a074256fc53eace134e WatchSource:0}: Error finding container 77add0aaa0a1dbfe6beaf093e38eab56a0a7ad669b0d1a074256fc53eace134e: Status 404 returned error can't find the container with id 77add0aaa0a1dbfe6beaf093e38eab56a0a7ad669b0d1a074256fc53eace134e Nov 03 23:23:44.400224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:44.400191 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" event={"ID":"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff","Type":"ContainerStarted","Data":"77add0aaa0a1dbfe6beaf093e38eab56a0a7ad669b0d1a074256fc53eace134e"} Nov 03 23:23:45.406166 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:45.406131 4903 generic.go:356] "Generic (PLEG): container finished" podID="87ee8fc7-1234-4e5b-a1e0-80420c88c0ff" containerID="21921d96f6ef0d2002902e290e6e8ce9126c3d2b1dfc5148db6882767e0b22b8" exitCode=0 Nov 03 23:23:45.406674 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:45.406225 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" event={"ID":"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff","Type":"ContainerDied","Data":"21921d96f6ef0d2002902e290e6e8ce9126c3d2b1dfc5148db6882767e0b22b8"} Nov 03 23:23:46.551623 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.551598 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:46.654011 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.653984 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-kube-api-access-76k9s\" (UniqueName: \"kubernetes.io/projected/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-kube-api-access-76k9s\") pod \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " Nov 03 23:23:46.654191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.654053 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-xks-ca\") pod \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " Nov 03 23:23:46.654191 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.654086 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-openshift-ca\") pod \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\" (UID: \"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff\") " Nov 03 23:23:46.654551 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.654517 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-openshift-ca" pod "87ee8fc7-1234-4e5b-a1e0-80420c88c0ff" (UID: "87ee8fc7-1234-4e5b-a1e0-80420c88c0ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:23:46.656256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.656236 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-kube-api-access-76k9s" pod "87ee8fc7-1234-4e5b-a1e0-80420c88c0ff" (UID: "87ee8fc7-1234-4e5b-a1e0-80420c88c0ff"). InnerVolumeSpecName "kube-api-access-76k9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:23:46.656256 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.656243 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-xks-ca" pod "87ee8fc7-1234-4e5b-a1e0-80420c88c0ff" (UID: "87ee8fc7-1234-4e5b-a1e0-80420c88c0ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:23:46.755794 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.755761 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76k9s\" (UniqueName: \"kubernetes.io/projected/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-kube-api-access-76k9s\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:46.755794 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.755788 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:46.755794 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:46.755797 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ee8fc7-1234-4e5b-a1e0-80420c88c0ff-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:47.416854 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:47.416817 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" event={"ID":"87ee8fc7-1234-4e5b-a1e0-80420c88c0ff","Type":"ContainerDied","Data":"77add0aaa0a1dbfe6beaf093e38eab56a0a7ad669b0d1a074256fc53eace134e"} Nov 03 23:23:47.416854 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:47.416849 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rvntn" Nov 03 23:23:47.417079 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:47.416866 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77add0aaa0a1dbfe6beaf093e38eab56a0a7ad669b0d1a074256fc53eace134e" Nov 03 23:23:48.383832 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:23:48.383796 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:23:54.642288 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.642251 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv"] Nov 03 23:23:54.642836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.642689 4903 cpu_manager.go:414] "containerMap: removing container" podUID="87ee8fc7-1234-4e5b-a1e0-80420c88c0ff" containerName="revoke-keys" Nov 03 23:23:54.642836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.642701 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ee8fc7-1234-4e5b-a1e0-80420c88c0ff" containerName="revoke-keys" Nov 03 23:23:54.642836 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.642793 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ee8fc7-1234-4e5b-a1e0-80420c88c0ff" containerName="revoke-keys" Nov 03 23:23:54.645993 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.645969 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.660038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.660005 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 23:23:54.660038 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.660023 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 23:23:54.660224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.660050 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 23:23:54.660224 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.660118 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7gjww\"" Nov 03 23:23:54.675764 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.675737 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv"] Nov 03 23:23:54.729947 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.729914 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n6hsv\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.730141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.730029 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lp8\" (UniqueName: \"kubernetes.io/projected/905d1adc-f14e-4d4a-8b1b-647d5037e41e-kube-api-access-j6lp8\") pod \"maas-api-revoke-sub-6b19338f-n6hsv\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.730141 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.730072 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n6hsv\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.831686 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.831652 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n6hsv\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.831898 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.831718 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lp8\" (UniqueName: \"kubernetes.io/projected/905d1adc-f14e-4d4a-8b1b-647d5037e41e-kube-api-access-j6lp8\") pod \"maas-api-revoke-sub-6b19338f-n6hsv\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.831898 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.831744 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n6hsv\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.832491 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.832462 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n6hsv\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.834257 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.834222 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n6hsv\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.853167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.853144 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lp8\" (UniqueName: \"kubernetes.io/projected/905d1adc-f14e-4d4a-8b1b-647d5037e41e-kube-api-access-j6lp8\") pod \"maas-api-revoke-sub-6b19338f-n6hsv\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:54.959094 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:54.959063 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:55.256374 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:55.256284 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv"] Nov 03 23:23:55.259660 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:23:55.259630 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905d1adc_f14e_4d4a_8b1b_647d5037e41e.slice/crio-f985c40e464f5584ff6c805dbf8ccc705b97b68a281517483f2a9ee9b005cf7e WatchSource:0}: Error finding container f985c40e464f5584ff6c805dbf8ccc705b97b68a281517483f2a9ee9b005cf7e: Status 404 returned error can't find the container with id f985c40e464f5584ff6c805dbf8ccc705b97b68a281517483f2a9ee9b005cf7e Nov 03 23:23:55.455873 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:55.455786 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" event={"ID":"905d1adc-f14e-4d4a-8b1b-647d5037e41e","Type":"ContainerStarted","Data":"f985c40e464f5584ff6c805dbf8ccc705b97b68a281517483f2a9ee9b005cf7e"} Nov 03 23:23:56.461770 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:56.461738 4903 generic.go:356] "Generic (PLEG): container finished" podID="905d1adc-f14e-4d4a-8b1b-647d5037e41e" containerID="7669786cdfa0952fca23e19158d40c8235101bc0f9a74290af8d21c7662a4139" exitCode=0 Nov 03 23:23:56.462166 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:56.461829 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" event={"ID":"905d1adc-f14e-4d4a-8b1b-647d5037e41e","Type":"ContainerDied","Data":"7669786cdfa0952fca23e19158d40c8235101bc0f9a74290af8d21c7662a4139"} Nov 03 23:23:57.604203 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.604176 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:57.762975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.762898 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-xks-ca\") pod \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " Nov 03 23:23:57.762975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.762953 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-openshift-ca\") pod \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " Nov 03 23:23:57.763219 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.762987 4903 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905d1adc-f14e-4d4a-8b1b-647d5037e41e-kube-api-access-j6lp8\" (UniqueName: \"kubernetes.io/projected/905d1adc-f14e-4d4a-8b1b-647d5037e41e-kube-api-access-j6lp8\") pod \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\" (UID: \"905d1adc-f14e-4d4a-8b1b-647d5037e41e\") " Nov 03 23:23:57.763361 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.763336 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-openshift-ca" pod "905d1adc-f14e-4d4a-8b1b-647d5037e41e" (UID: "905d1adc-f14e-4d4a-8b1b-647d5037e41e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 23:23:57.763469 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.763452 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-openshift-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:57.765192 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.765156 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-xks-ca" pod "905d1adc-f14e-4d4a-8b1b-647d5037e41e" (UID: "905d1adc-f14e-4d4a-8b1b-647d5037e41e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 23:23:57.765333 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.765227 4903 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905d1adc-f14e-4d4a-8b1b-647d5037e41e-kube-api-access-j6lp8" pod "905d1adc-f14e-4d4a-8b1b-647d5037e41e" (UID: "905d1adc-f14e-4d4a-8b1b-647d5037e41e"). InnerVolumeSpecName "kube-api-access-j6lp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 23:23:57.864980 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.864943 4903 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6lp8\" (UniqueName: \"kubernetes.io/projected/905d1adc-f14e-4d4a-8b1b-647d5037e41e-kube-api-access-j6lp8\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:57.864980 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:57.864982 4903 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905d1adc-f14e-4d4a-8b1b-647d5037e41e-maas-api-xks-ca\") on node \"ip-10-0-133-193.ec2.internal\" DevicePath \"\"" Nov 03 23:23:58.473606 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:58.473568 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" event={"ID":"905d1adc-f14e-4d4a-8b1b-647d5037e41e","Type":"ContainerDied","Data":"f985c40e464f5584ff6c805dbf8ccc705b97b68a281517483f2a9ee9b005cf7e"} Nov 03 23:23:58.473606 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:58.473593 4903 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n6hsv" Nov 03 23:23:58.473838 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:23:58.473623 4903 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f985c40e464f5584ff6c805dbf8ccc705b97b68a281517483f2a9ee9b005cf7e" Nov 03 23:24:01.899557 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:01.899522 4903 scope.go:122] "RemoveContainer" containerID="3eb0780ccf9dde10d400ffc5f0afb20e4ef45080e4df6b382821d95d959e2688" Nov 03 23:24:05.919607 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:05.919572 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dg6s6"] Nov 03 23:24:06.082975 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:06.082883 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59cc64bfd5-wtjfp_f374375e-68d7-4898-9a67-c4bb3b1be125/authorino/0.log" Nov 03 23:24:18.384285 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:24:18.384246 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:24:19.241271 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:19.241243 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rvntn_87ee8fc7-1234-4e5b-a1e0-80420c88c0ff/revoke-keys/0.log" Nov 03 23:24:19.789249 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:19.789211 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-n6hsv_905d1adc-f14e-4d4a-8b1b-647d5037e41e/revoke-keys/0.log" Nov 03 23:24:20.133337 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:20.133244 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lgbpb_84ccae5c-0781-4ddb-9422-8d8dcae660cc/revoke-keys/0.log" Nov 03 23:24:21.784901 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:21.784866 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59cc64bfd5-wtjfp_f374375e-68d7-4898-9a67-c4bb3b1be125/authorino/0.log" Nov 03 23:24:22.893244 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:22.893216 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dg6s6_7a093438-7423-4f79-903b-b88a4e05b08c/limitador/0.log" Nov 03 23:24:24.666552 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:24.666525 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b75ff7d64-dqsnv_69f95d94-ca0e-475b-b197-956d4dc9556a/kube-auth-proxy/0.log" Nov 03 23:24:24.866486 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:24.866459 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-f7pll_8e3a38ee-6f79-46eb-a285-8052c4b0228f/istio-proxy/0.log" Nov 03 23:24:25.044346 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:25.044261 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-fkpql_08263e96-b3ee-4661-9110-4a06dcae57e3/istio-proxy/0.log" Nov 03 23:24:26.221022 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:26.220994 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-xt8wx_534555a5-ce83-4e60-a28a-b410824419ac/istio-proxy/0.log" Nov 03 23:24:26.970255 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:26.970227 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-wnlt4_5d9af6c0-52ea-4612-8d1f-b3d4138c8f32/payload-pre-processing/0.log" Nov 03 23:24:27.156987 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:27.156960 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:24:27.361024 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:27.360952 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68496b87c-6xdvq_0a18566a-0818-494c-989b-da2abea7edd9/router/0.log" Nov 03 23:24:28.168492 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:28.168458 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bw949_29f56bb8-a11b-4b4a-86b0-68f9c535f694/main/0.log" Nov 03 23:24:28.569227 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:28.569159 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2z58q_a8b9b4d3-1dde-4142-84e9-3391d26a3b1c/main/0.log" Nov 03 23:24:28.944987 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:28.944956 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7_c9d72039-62c4-4d01-85f9-4e65d431a4a8/main/0.log" Nov 03 23:24:29.324696 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:29.324619 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t_dbc38553-698d-4870-9a3a-cb198e6dc530/main/0.log" Nov 03 23:24:29.512615 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:29.512583 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h_c282635b-b4c8-40f8-880f-a773dc994c41/main/0.log" Nov 03 23:24:30.395387 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:30.395358 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59cc64bfd5-wtjfp_f374375e-68d7-4898-9a67-c4bb3b1be125/authorino/0.log" Nov 03 23:24:41.749303 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:41.749263 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rvntn_87ee8fc7-1234-4e5b-a1e0-80420c88c0ff/revoke-keys/0.log" Nov 03 23:24:42.120353 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:42.120249 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-n6hsv_905d1adc-f14e-4d4a-8b1b-647d5037e41e/revoke-keys/0.log" Nov 03 23:24:42.496341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:42.496288 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lgbpb_84ccae5c-0781-4ddb-9422-8d8dcae660cc/revoke-keys/0.log" Nov 03 23:24:43.819136 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:43.819105 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59cc64bfd5-wtjfp_f374375e-68d7-4898-9a67-c4bb3b1be125/authorino/0.log" Nov 03 23:24:44.899920 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:44.899891 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dg6s6_7a093438-7423-4f79-903b-b88a4e05b08c/limitador/0.log" Nov 03 23:24:46.274085 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:46.274054 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b75ff7d64-dqsnv_69f95d94-ca0e-475b-b197-956d4dc9556a/kube-auth-proxy/0.log" Nov 03 23:24:46.465180 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:46.465149 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-f7pll_8e3a38ee-6f79-46eb-a285-8052c4b0228f/istio-proxy/0.log" Nov 03 23:24:46.653682 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:46.653653 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-fkpql_08263e96-b3ee-4661-9110-4a06dcae57e3/istio-proxy/0.log" Nov 03 23:24:47.864864 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:47.864832 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-xt8wx_534555a5-ce83-4e60-a28a-b410824419ac/istio-proxy/0.log" Nov 03 23:24:48.383630 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:24:48.383591 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:24:48.648096 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:48.648018 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-wnlt4_5d9af6c0-52ea-4612-8d1f-b3d4138c8f32/payload-pre-processing/0.log" Nov 03 23:24:48.824888 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:48.824860 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:24:49.009519 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:49.009487 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68496b87c-6xdvq_0a18566a-0818-494c-989b-da2abea7edd9/router/0.log" Nov 03 23:24:49.546283 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:49.546250 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bw949_29f56bb8-a11b-4b4a-86b0-68f9c535f694/main/0.log" Nov 03 23:24:49.904643 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:49.904617 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2z58q_a8b9b4d3-1dde-4142-84e9-3391d26a3b1c/main/0.log" Nov 03 23:24:50.268066 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:50.267994 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8f8hs7_c9d72039-62c4-4d01-85f9-4e65d431a4a8/main/0.log" Nov 03 23:24:50.634500 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:50.634423 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-rr82t_dbc38553-698d-4870-9a3a-cb198e6dc530/main/0.log" Nov 03 23:24:50.818407 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:24:50.818378 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-8776h_c282635b-b4c8-40f8-880f-a773dc994c41/main/0.log" Nov 03 23:25:01.926595 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:25:01.926558 4903 scope.go:122] "RemoveContainer" containerID="4d8ed773180e613d4ee4fee524b663cf9dfedbcae79f7f0bbefb69ec8390991a" Nov 03 23:25:01.934970 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:25:01.934947 4903 scope.go:122] "RemoveContainer" containerID="93161df3ddd0b0f6cff88125a2b2623c0e0776c475a465ad344bb79d2951af5b" Nov 03 23:25:18.383427 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:25:18.383377 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:25:26.578516 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:25:26.578485 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:25:26.767821 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:25:26.767795 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:25:48.382986 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:25:48.382901 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:26:10.985679 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:10.985648 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:26:11.169421 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.169394 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-bnqlr_76f24aa5-5e88-488f-90e3-252e1e114b39/payload-processing/0.log" Nov 03 23:26:11.479027 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.478992 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qtpzb/must-gather-9l8lb"] Nov 03 23:26:11.479420 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.479400 4903 cpu_manager.go:414] "containerMap: removing container" podUID="905d1adc-f14e-4d4a-8b1b-647d5037e41e" containerName="revoke-keys" Nov 03 23:26:11.479502 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.479421 4903 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905d1adc-f14e-4d4a-8b1b-647d5037e41e" containerName="revoke-keys" Nov 03 23:26:11.479545 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.479512 4903 memory_manager.go:365] "RemoveStaleState removing state" podUID="905d1adc-f14e-4d4a-8b1b-647d5037e41e" containerName="revoke-keys" Nov 03 23:26:11.482627 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.482611 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtpzb/must-gather-9l8lb" Nov 03 23:26:11.505874 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.505846 4903 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qtpzb\"/\"default-dockercfg-8d4gm\"" Nov 03 23:26:11.506021 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.505953 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qtpzb\"/\"kube-root-ca.crt\"" Nov 03 23:26:11.506021 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.505956 4903 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qtpzb\"/\"openshift-service-ca.crt\"" Nov 03 23:26:11.514676 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.514651 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtpzb/must-gather-9l8lb"] Nov 03 23:26:11.548935 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.548900 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8c0a063c-8f1b-4fd2-a43e-d96fd93f5173-must-gather-output\") pod \"must-gather-9l8lb\" (UID: \"8c0a063c-8f1b-4fd2-a43e-d96fd93f5173\") " pod="openshift-must-gather-qtpzb/must-gather-9l8lb" Nov 03 23:26:11.549102 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.548942 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv9vl\" (UniqueName: \"kubernetes.io/projected/8c0a063c-8f1b-4fd2-a43e-d96fd93f5173-kube-api-access-pv9vl\") pod \"must-gather-9l8lb\" (UID: \"8c0a063c-8f1b-4fd2-a43e-d96fd93f5173\") " pod="openshift-must-gather-qtpzb/must-gather-9l8lb" Nov 03 23:26:11.649542 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.649506 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8c0a063c-8f1b-4fd2-a43e-d96fd93f5173-must-gather-output\") pod \"must-gather-9l8lb\" (UID: \"8c0a063c-8f1b-4fd2-a43e-d96fd93f5173\") " pod="openshift-must-gather-qtpzb/must-gather-9l8lb" Nov 03 23:26:11.649758 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.649549 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv9vl\" (UniqueName: \"kubernetes.io/projected/8c0a063c-8f1b-4fd2-a43e-d96fd93f5173-kube-api-access-pv9vl\") pod \"must-gather-9l8lb\" (UID: \"8c0a063c-8f1b-4fd2-a43e-d96fd93f5173\") " pod="openshift-must-gather-qtpzb/must-gather-9l8lb" Nov 03 23:26:11.649932 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.649909 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8c0a063c-8f1b-4fd2-a43e-d96fd93f5173-must-gather-output\") pod \"must-gather-9l8lb\" (UID: \"8c0a063c-8f1b-4fd2-a43e-d96fd93f5173\") " pod="openshift-must-gather-qtpzb/must-gather-9l8lb" Nov 03 23:26:11.670870 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.670842 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv9vl\" (UniqueName: \"kubernetes.io/projected/8c0a063c-8f1b-4fd2-a43e-d96fd93f5173-kube-api-access-pv9vl\") pod \"must-gather-9l8lb\" (UID: \"8c0a063c-8f1b-4fd2-a43e-d96fd93f5173\") " pod="openshift-must-gather-qtpzb/must-gather-9l8lb" Nov 03 23:26:11.796120 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:11.796038 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtpzb/must-gather-9l8lb" Nov 03 23:26:12.094728 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:12.094590 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtpzb/must-gather-9l8lb"] Nov 03 23:26:12.097088 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:26:12.097051 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c0a063c_8f1b_4fd2_a43e_d96fd93f5173.slice/crio-a738b1a9cd43449c54ffb9090a879149e9b652e477c49e97e1bc1ff159f5b089 WatchSource:0}: Error finding container a738b1a9cd43449c54ffb9090a879149e9b652e477c49e97e1bc1ff159f5b089: Status 404 returned error can't find the container with id a738b1a9cd43449c54ffb9090a879149e9b652e477c49e97e1bc1ff159f5b089 Nov 03 23:26:12.098901 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:12.098882 4903 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 23:26:12.128551 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:12.128510 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtpzb/must-gather-9l8lb" event={"ID":"8c0a063c-8f1b-4fd2-a43e-d96fd93f5173","Type":"ContainerStarted","Data":"a738b1a9cd43449c54ffb9090a879149e9b652e477c49e97e1bc1ff159f5b089"} Nov 03 23:26:13.134333 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:13.134262 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtpzb/must-gather-9l8lb" event={"ID":"8c0a063c-8f1b-4fd2-a43e-d96fd93f5173","Type":"ContainerStarted","Data":"bea3eedc6fa694c04f04507c0b3a6f58b4daf22d3d6428b5f4e429da3dd814bf"} Nov 03 23:26:13.134333 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:13.134325 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtpzb/must-gather-9l8lb" event={"ID":"8c0a063c-8f1b-4fd2-a43e-d96fd93f5173","Type":"ContainerStarted","Data":"bbcad9c77d8b9186ac6612db29aa981b16a43b21ed8bd796c625f03628124633"} Nov 03 23:26:13.171957 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:13.171902 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qtpzb/must-gather-9l8lb" podStartSLOduration=1.396635326 podStartE2EDuration="2.17188486s" totalImagesPullingTime="775.249534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 23:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:26:13.170789935 +0000 UTC m=+4399.574730903" watchObservedRunningTime="2026-11-03 23:26:13.17188486 +0000 UTC m=+4399.575825836" Nov 03 23:26:16.415621 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:16.415593 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vk57x_05ab17d1-b5dc-4cb6-bf8e-eb650c724aef/global-pull-secret-syncer/0.log" Nov 03 23:26:16.599052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:16.599025 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-grdpd_63867b60-8f1a-44a6-bbf1-8e521d232081/connection-checker/0.log" Nov 03 23:26:16.724866 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:16.724794 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-l44xs_a714258f-ba4e-4792-853b-7d20411d3ca7/connection-checker/0.log" Nov 03 23:26:16.853268 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:16.853146 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-t67lk_7c35b7b2-f209-4ff3-ab43-d8533fcdf341/connection-checker/0.log" Nov 03 23:26:17.353341 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:17.353302 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d4tnh_e881588e-ae98-4f04-947d-1a765b8a4e5e/konnectivity-agent/0.log" Nov 03 23:26:17.778118 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:17.778080 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-193.ec2.internal_ebb54ef3d211c3a17d214f74e032fea7/haproxy/0.log" Nov 03 23:26:18.423099 ip-10-0-133-193 kubenswrapper[4903]: E1103 23:26:18.423050 4903 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7vs67" containerName="extractor" Nov 03 23:26:24.407966 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:24.407930 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59cc64bfd5-wtjfp_f374375e-68d7-4898-9a67-c4bb3b1be125/authorino/0.log" Nov 03 23:26:25.053345 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:25.050769 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dg6s6_7a093438-7423-4f79-903b-b88a4e05b08c/limitador/0.log" Nov 03 23:26:26.255876 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.255837 4903 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2"] Nov 03 23:26:26.260240 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.260211 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.287065 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.287029 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2"] Nov 03 23:26:26.325855 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.325807 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-proc\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.326057 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.325899 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5kt\" (UniqueName: \"kubernetes.io/projected/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-kube-api-access-2b5kt\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.326057 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.325977 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-podres\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.326057 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.326022 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-sys\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.326057 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.326047 4903 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-lib-modules\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.427441 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.427387 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-podres\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.427648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.427466 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-sys\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.427648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.427491 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-lib-modules\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.427648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.427543 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-proc\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.427648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.427553 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-sys\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.427648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.427541 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-podres\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.427648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.427618 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-lib-modules\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.427648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.427629 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-proc\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.427648 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.427624 4903 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5kt\" (UniqueName: \"kubernetes.io/projected/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-kube-api-access-2b5kt\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.448807 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.448776 4903 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5kt\" (UniqueName: \"kubernetes.io/projected/c70e1048-3064-4c81-9d53-bf09e4b0a3ee-kube-api-access-2b5kt\") pod \"perf-node-gather-daemonset-fjkl2\" (UID: \"c70e1048-3064-4c81-9d53-bf09e4b0a3ee\") " pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.581093 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.581010 4903 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:26.894811 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:26.894729 4903 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2"] Nov 03 23:26:26.898329 ip-10-0-133-193 kubenswrapper[4903]: W1103 23:26:26.898275 4903 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc70e1048_3064_4c81_9d53_bf09e4b0a3ee.slice/crio-01c52610206c5e2b4cd26f7ca135594fd4338bfaa7d20ef93abaf044b06b32a3 WatchSource:0}: Error finding container 01c52610206c5e2b4cd26f7ca135594fd4338bfaa7d20ef93abaf044b06b32a3: Status 404 returned error can't find the container with id 01c52610206c5e2b4cd26f7ca135594fd4338bfaa7d20ef93abaf044b06b32a3 Nov 03 23:26:27.308023 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:27.307982 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" event={"ID":"c70e1048-3064-4c81-9d53-bf09e4b0a3ee","Type":"ContainerStarted","Data":"fae4971804330bec7f0fb5fdf2ef7b548a1cfadda729c9917d03834359404527"} Nov 03 23:26:27.308534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:27.308063 4903 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" event={"ID":"c70e1048-3064-4c81-9d53-bf09e4b0a3ee","Type":"ContainerStarted","Data":"01c52610206c5e2b4cd26f7ca135594fd4338bfaa7d20ef93abaf044b06b32a3"} Nov 03 23:26:27.308534 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:27.308085 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:27.344052 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:27.343989 4903 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" podStartSLOduration=1.343972347 podStartE2EDuration="1.343972347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 23:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 23:26:27.343880277 +0000 UTC m=+4413.747821252" watchObservedRunningTime="2026-11-03 23:26:27.343972347 +0000 UTC m=+4413.747913322" Nov 03 23:26:27.533998 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:27.533963 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-v9lq6_cf0d1c58-9465-423d-b479-50956f62c7f2/cluster-monitoring-operator/0.log" Nov 03 23:26:28.266250 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:28.266218 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2vzkk_2e77d4d0-4aba-43cc-92df-7f9073838727/node-exporter/0.log" Nov 03 23:26:28.389866 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:28.389836 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2vzkk_2e77d4d0-4aba-43cc-92df-7f9073838727/kube-rbac-proxy/0.log" Nov 03 23:26:28.514546 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:28.514518 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2vzkk_2e77d4d0-4aba-43cc-92df-7f9073838727/init-textfile/0.log" Nov 03 23:26:33.329396 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:33.329374 4903 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qtpzb/perf-node-gather-daemonset-fjkl2" Nov 03 23:26:33.455523 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:33.455480 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-zrq7h_c9994b27-e2c7-4e4a-ab0c-796c19becb5c/networking-console-plugin/0.log" Nov 03 23:26:34.648458 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:34.648425 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/1.log" Nov 03 23:26:34.652331 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:34.652247 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-xsr82_0c3d06f5-c335-4cf2-a901-cae1f6c2e142/console-operator/2.log" Nov 03 23:26:36.967794 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:36.967759 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-fgqk2_305d6b5c-15d1-4c7e-8b31-db2c71402cd3/volume-data-source-validator/0.log" Nov 03 23:26:39.777233 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:39.777200 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8vfzp_e302a1bc-43db-4e8b-8ed4-f42194f5ddeb/dns/0.log" Nov 03 23:26:39.901234 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:39.901202 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8vfzp_e302a1bc-43db-4e8b-8ed4-f42194f5ddeb/kube-rbac-proxy/0.log" Nov 03 23:26:40.029913 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:40.029831 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7b4rf_677c391f-bc16-46c8-9439-02a1f3bf934d/dns-node-resolver/0.log" Nov 03 23:26:41.519167 ip-10-0-133-193 kubenswrapper[4903]: I1103 23:26:41.519111 4903 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctz65_c1b7053e-001b-4d96-9bf3-1a768e364346/node-ca/0.log"